Tracking Corrupt Cortex XDR Agents

[thenetworksfine]

I am looking for any input on how other customers are handling situations where:

1. The agent is installed on a host and says it is checking in, but it does not appear in the Cortex XDR Console

2. The agent is corrupt and has stopped reporting back (due to a failed upgrade or otherwise)

I didn't know if anyone has any unique solutions for these situations. From a corrupt agent standpoint, it would be nice to have a Tenable plugin to report back the current signature versions.

[Alexandre_Jodoin]

I can't say that I saw 1) but 2) sounds too familliar.

I estimate that we have, at any given time, about 3-5% of the agents in that situation.

Basically, I have a script that will re-create/re-enable the runtimes, as I found this is mostly the case with corrupted installation.

The cyserver services won't start as one of the dependencies is borked. So the script tries to fix them all.

This works in 20% of the corrupted agent cases.

For the others it is an issue with the file system filter. trying to reload it using fltmc fails with a "file not found" error and I haven't been able to find a solution for those, so it is xdrcleaner, reboot, xdrcleaner, re-install.

But 3-5% of 18k agents is a lot to manually fix month after month.

[btenberge]

I'm sorry I don't have an answer to your question either, but I think this is a good subject because I'm having the same "revelations" lately.

I have encountered some agents in our environment that appear to be working (they appear online and up-to-date), but half the files are missing in the installation folder, only cyserver.exe is running and upgrade attempts fail all the time.

These specific instances do make it seem like everything is fine and dandy, while it's not.

I'm currently trying to figure out a way to find these half-, or non-operational agents, myself, but I have nothing yet.

I'm sure we're not the only ones dealing with this problem, so I'm hoping that there are some people that have already found a working solution.

[thenetworksfine]

@Alexandre_Jodoin We are working through a wide range of hosts with support to get to the bottom of this situation. One of the commonalities ended up being deleted Installation Packages under Cortex XDR Administrative Console >> Endpoints >> Endpoint Management >> Agent Installations. I'm not saying this is the case for you but we did not know that someone was cleaning up these packages and wouldn't have though it would put the agent dead in the water. The agent is not smart enough to go out and get the latest version from the console if the previously sent version no longer exists.

A helpful command that we used is below. This prevented us from having to reinstall the agent in a number of situations.

1. From an administrative command prompt
   1. C:\Program Files\Palo Alto Networks\Traps\cytool.exe reconnect force <ID>
2. Where <ID> = the agent ID that corresponds to an agent installation package name with the installed version of the agent. You can show the id field within Cortex XDR Administrative Console >> Endpoints >> Endpoint Management >> Agent Installations.
3. The agent administrative password is then required
4. Click Check In Now on the agent console

We are working through some other scenarios so I will update this post accordingly in hopes to help out some other customers in our situation.

[thenetworksfine]

@btenberge See my reply to @Alexandre_Jodoin below.

[Alexandre_Jodoin]

Thank you for this, it is an option I wasn't aware of, and I will give this a try on a few agents to see.

When trying to delete and installation package from the console it DOES give a warning:"This will prevent new agents using the package, including VDI, from registering."

So I've never deleted a single package. Not sure if that will be an issue 5 years from now... 

Our main issue has to do with the file system filter driver/main cyserver service.

Here's a bacth file I created to help with service issues. However this does NOT fix the cyvrfsfd filter issue we are seing.

NOT the silver bullet but might help to re-enable some agents...

-----------------------

@echo on

REM Stop the runtimes
echo PasswordHERE!!!|"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" runtime stop

REM remove autoprotection
echo PasswordHERE!!!|"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" protect disable

sc create cyvrfsfd binpath= "C:\Program Files\Palo Alto Networks\Traps\cyvrfsfd.sys" type= filesys start= system error= normal group= "FSFilter Anti-Virus" tag= yes displayname= cyvrfsfd depend= FltMgr || sc config cyvrfsfd binpath= "C:\Program Files\Palo Alto Networks\Traps\cyvrfsfd.sys" type= filesys start= system error= normal group= "FSFilter Anti-Virus" tag= yes displayname= cyvrfsfd depend= FltMgr
sc create cyverak type= KERNEL start= system error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\cyverak.sys" tag= no displayname= cyverak || sc config cyverak type= KERNEL start= system error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\cyverak.sys" tag= no displayname= cyverak
sc create cyvrmtgn type= KERNEL start= system error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\cyvrmtgn.sys" tag= no displayname= cyvrmtgn || sc config cyvrmtgn type= KERNEL start= system error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\cyvrmtgn.sys" tag= no displayname= cyvrmtgn
sc create tedrdrv type= filesys start= system error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\tedrdrv.sys" group= "FSFilter Activity Monitor" tag= yes displayname= tedrdrv depend= FltMgr || sc config tedrdrv type= filesys start= system error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\tedrdrv.sys" group= "FSFilter Activity Monitor" tag= yes displayname= tedrdrv depend= FltMgr
sc create cyserver type= own start= auto error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\cyserver.exe" tag= no displayname= "Cortex XDR" depend= Cyvrmtgn/Cyverak/Cyvrfsfd/EventLog/CryptSvc/TEdrDrv || sc config cyserver type= own start= auto error= normal binpath= "C:\Program Files\Palo Alto Networks\Traps\cyserver.exe" tag= no displayname= "Cortex XDR" depend= Cyvrmtgn/Cyverak/Cyvrfsfd/EventLog/CryptSvc/TEdrDrv

REM Oh CRAP! sc start ftlmgr -> The specified service does not exist as an installed service
sc config FltMgr type= filesys start= boot error= critical binpath= "C:\Windows\system32\drivers\fltmgr.sys" group= "FSFilter Infrastructure" tag= yes displayname= FltMgr || sc create FltMgr type= filesys start= boot error= critical binpath= "C:\Windows\system32\drivers\fltmgr.sys" group= "FSFilter Infrastructure" tag= yes displayname= FltMgr

REM Set the services/drivers/filters to auto start, not required since we did it already.
REM sc config cyserver start= auto && sc config cyverak start=system && sc config cyvrmtgn start= system && sc config cyvrfsfd start= system && sc config tedrdrv start= system

REM Enable autoprotection
echo PasswordHERE!!!|"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" protect enable

REM start the services/drivers/filters. Use one or the other. Or both.
sc start cyverak & sc start cyvrmtgn & sc start cyvrfsfd & sc start tedrdrv & sc start cyserver
REM "C:\Program Files\Palo Alto Networks\Traps\cytool.exe" runtime start

REM Try reconnecting if communication with server has been disabled
echo PasswordHERE!!!|"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" reconnect

REM Update Cortex XDR from server
"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" checkin

REM Check Protection determined by policy
REM echo PasswordHERE!!!|"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" protect policy

REM Query product components running state
"C:\Program Files\Palo Alto Networks\Traps\cytool.exe" runtime query

REM Altitude and volumes for the filter can be found by issuing this command on an agent in working order:

REM fltmc instances

REM Filtre Nom du volume Altitude Nom de l’instance Cadre SprtFtrs VlStatus
REM -------------------- ------------------------------------- ------------ ---------------------- ----- -------- --------
REM cyvrfsfd 321234 Cyvera FSFD 0 00000007
REM cyvrfsfd C: 321234 Cyvera FSFD 0 00000007
REM cyvrfsfd 321234 Cyvera FSFD 0 00000007
REM cyvrfsfd \Device\Mup 321234 Cyvera FSFD 0 00000007

REM Saw this one once. Not bother fixing it for all. But here's the info if required.

REM SERVICE_NAME: telam
REM TYPE : 1 KERNEL_DRIVER
REM START_TYPE : 0 BOOT_START
REM ERROR_CONTROL : 1 NORMAL
REM BINARY_PATH_NAME : \SystemRoot\system32\drivers\telam.sys
REM LOAD_ORDER_GROUP : early-launch
REM TAG : 0
REM DISPLAY_NAME : telam
REM DEPENDENCIES :
REM SERVICE_START_NAME :

REM sc create telam binpath= "C:\Windows\system32\drivers\telam.sys" type= kernel start= boot error= normal group= "early-launch" tag= no displayname= telam

[btenberge]

@thenetworksfine Thanks for your reply!

I haven't ever removed any old installation packages, I think, but it's worth checking out for sure.

[btenberge]

Your reply could also be quite helpful, @Alexandre_Jodoin, thanks for sharing your solution!

You guys are giving me some interesting things to look at.

[MParker4]

Does anyone have an update to this issue, it seems to exist around version upgrades that a small percent will stop responding.

[bbarmanroy]

Hi @MParker4 , this seems like an issue with the agent. Please generate a support file for a few affected endpoints, create a Support Ticket, and upload the support files for the PANW TAC Engineers to investigate.

[thenetworksfine]

Hopefully this will save you from months of log collection. There is a known bug with agent upgrades prior to the versions below. Unfortunately all support reps are not aware. See the comment below from support that we received once the right people got involved with our ticket.

"Moving forward from 7.4.1->7.4.2 and onward you should not run into this issue"

What version are you noticing issues with?