Performance Optimization Strategies

[JesseBaxter]

Hello,

I have a system running workloads that is sensitive to CPU usage. The primary users have raised an issue that our currently applied XDR profile is causing detrimental delays in job processing. I confirmed this by disabling the agent to observe baseline activity. The current profile/policy has Exploit and Malware prevention set to "monitor only'. 

Posting here hoping for some recommendations how to address the performance issues, as I'm not the most familiar with Cortex XDR yet. The system pulls reports from internal NAS, combines them using a variety of batch jobs, and sends them off. The user provided the following description when I inquired about process exclusions: 

As far as "processes" go, the primary processes are the Poster process, which is a number of batch scripts and PosterWatcher.exe. The scripts that make up the batch processing system, Poster.cmd and PostForm.cmd primarily use built-in commands and executables. It's also possible that they call MQRequestClient.exe. There is also NT.Combine.cmd which primarily also uses built-in commands. 

Given the context, what would be some appropriate strategies to exclude some of the sensitive workloads? I don't want to take a blunt approach such as whitelisting cmd.exe completely. 

Thanks in advance!
Jesse 

[aleksandar.astardzhiev]

Hi @JesseBaxter ,

The only confirmed case where XDR was causing high CPU for us, was due to the "Network Packet Inspection engine". We ended up disabling it for servers that require high amount of network traffic.

In addition below is a snipped from Palo TAC suggesting how to narrow down which module is causing the problem (for another case where we couldn't confirm XDR was the culprit)

5)Since XDR has multiple protection layers, we need to understand which module is causing delay issues. Once we understand the root cause we can collect the logs accordingly and provide the solution. 

 

>> Stop event collection by running this command and checking if there is a change in CPU usage? Take a screenshot of memory utilization. 

 

/opt/traps/bin/cytool event_collection disable 

*this will ask for the supervisor password which is also the agent uninstall password 

 

>> Disable Malware protection and verify if there is a change in CPU usage 

-create a new Malware Security Profile with all settings action mode set to disabled 

https://docs.paloaltonetworks.com/cortex/cortex-xdr/cortex-xdr-prevent-admin/endpoint-security/endpoint-security-profiles/add-malware-security-profile.html 

-apply malware security profile to the target machine 

-do a check-in on the Cortex XDR agent 

-check if there is a change in CPU usage. Take a screenshot of Cortex XDR CPU utilization. 

-restart Cortex XDR agent 

/opt/traps/bin/cytool runtime stop 

/opt/traps/bin/cytool runtime start 

-check Cortex XDR CPU usage. Is it going high immediately after starting the service? 

 

>> Disable Exploit protection and verify if the issue will be reproduced 

-create a new Exploit Security Profile with all settings action mode set to disabled 

https://docs.paloaltonetworks.com/cortex/cortex-xdr/cortex-xdr-prevent-admin/endpoint-security/endpoint-security-profiles/add-exploit-security-profile.html 

-apply to Exploit security profile to a target machine 

-do a check-in on the Cortex agent 

-check if there is a change in CPU usage. Take a screenshot of Cortex XDR CPU utilization. 

[JesseBaxter]

Thanks a lot for sharing. We are going to narrow down the performance problems using the strategies you outlined. 

[E.Jafarov]

Hi aleksandar.astardzhiev ,

For example, if I notice that the main high CPU usage is in malware profile, what should I do with malware profile?