Cortex-XDR

[Vishwasamudra]

Cortex-XDR agent rollback feature not available, its recommended for  deployment in large network.

Please note you are posting a public message where community members and experts can provide assistance. Sharing private information such as serial numbers or company information is not recommended.

[bbarmanroy]

Hi @Vishwasamudra what would be the reason for a rollback? My recommendation is to first deploy a new version of an agent on a set of UAT servers. Upon UAT validation, you can proceed to deploy on a wider set of endpoints. Irrespective of deployment sizes, this should be the process to ensure operational stability.

[Vishwasamudra]

This could not be a solution its one of the deployment practice/method in production environment. what are the reasons to add agent rollback feature. other end point protection have providing why cant they. 

[bbarmanroy]

Hi @Vishwasamudra it is not clear why you'd want to perform a rollback. Is it because of performance issues? If so, it'd be prudent to roll out a new version on UAT before production deployment. This would help identify any issues in UAT, work with support to fix the issue and proceed to production.

Secondly, a rollback exposes the endpoints to known issues that were discovered and documented on older versions. This is not an ideal solution. 

Thirdly, a rollback is not a long-term solution as the organization would still need to push an upgrade to ensure the agents do not reach EoL. Once an agent reaches EoL, it stops receiving Content Updates and exposes an endpoint to known vulnerabilities, which are detected/prevented by Content Updates.

Fourthly, a rollback to an older version removes the capability of the agent to have additional security modules which were introduced in newer versions.

Finally, a rollback is a last resort - which can be managed with software inventory tools like SCCM, Intune etc.

I look forward to hearing your points on the need for rollback and why my recommendations do not fit your need as a best practice.

[Vishwasamudra]

Hello,

Solution from Polo alto on end point security..

Cortex-XDR not provided auto upgrade with desired version for large scale of networks ( mostly 500 to 1000 systems)

auto upgrade will take latest patch what released soon...

Todays life every thing going to IOT/automation with less man power.

If my XDR profile set to auto update, need provision agent rollback with N+1.

[bbarmanroy]

Hi @Vishwasamudra it is not clear from your last post what you are suggesting. 

I will try to explain your concerns:

1. "Cortex-XDR not provided auto upgrade with desired version for large scale of networks ( mostly 500 to 1000 systems)" -  XDR does it via Agent profiles. You can also set it to a desired minor release version in your profiles. You can also use Action Center to restrict the deployment to a desired agent maintenance release. 

2. "Todays life every thing going to IOT/automation with less man power" - absolutely. Please leverage your organization's GPO/SCCM to deploy agents for large environments such as yours.

I look forward to your explanation on why you want to rollback to an older version (which exposes the environment to known issues), and what is preventing you from doing so with SCCM?

[RHolzer]

Hi @bbarmanroy! Maybe I can jump in as I have a similar need. Starting with CU 540-92670, our Windows 7/2008R2 clients with 7.7.x-agents installed used >=50% of CPUs of the machine. Machines with the 7.5 (CE) agent and the same CU didn't have this issue. The affected machines weren't usable at all so we were forced to react quick.

We "solved" the issue by uninstalling the agents via the management platform (offtopic: because we didn't have the current agent remover - is there an easy way to get the latest version or do I really have to ask the support for every version?) and redeploying 7.5 (CE) agents to affected systems as those are about to be replaced anyway. This has happened for the first time in two years of using Cortex XDR, but the ability to downgrade would have helped to save some time for sure. Only "a hand full" (< 100) of clients were affected, so the process was no big issue, but imagine this happening for > 1.000 clients. That would be a big pain! 😉

[eluis]

Hi @Vishwasamudra, 
it would be good to understand the reason for an agent version rollback. This action would open your systems to old security issues and attack vectors that are already blocked on newer versions. On top of that you might also come back to old versions performance issues or bugs that have been already solved. 

I agree with Bbarmanroy that the best practice is to have all your agents in the newest version and if you dont feel confident to move to the very latest for some critical endpoints, you can upgrade in UAT and once you feel confident you can  move on with all your endpoints upgrade. 

KR,
Luis

[bbarmanroy]

Hi @RHolzer Thank you for your point. I can understand the challenges you went through. What you've explained is an issue with the CU. However, I'd like to reiterate the same process for CU rollout as with agent upgrades - consider a strategy for upgrading your CU's in a delayed fashion for your PROD environments after UAT validation. The issue with CU 540-92670 was resolved with 540-92744. Rolling back to a CE version claws back the new features that were introduced in the later versions. You should consider using it in critical environments only: "Critical Environment Versions are designed for sensitive and highly regulated environments and do not contain all updates and content existing in the standard version. Therefore, it is recommended to restrict the use of these versions to the required minimum."  Ref: https://docs.paloaltonetworks.com/cortex/cortex-xdr/cortex-xdr-pro-admin/endpoint-security/manage-co...

That being said, we're discussing internally to see how we can let users control the CU version.

[RHolzer]

Hi @bbarmanroy would have the delay period actually helped in this case? In my opinion there are two possibilitys when delaying the CUs, for example the delay is 1 day:

1. The faulty 540-92670 is being released and after 1 day it is being introduced to our environment. The 540-92744 is being released some hours later and also after 1 day (but some hours later?) it is being introduced and superseeds the older 540-92744 CU. In thise case, the CU was delayed, but we would have had the same issue.

2. The 540-92670 is being released and after 1 day it would have been introduced to our environment, but it's already superseeded by 540-92744, so 1 day later, we will get the 540-92744 and skip the 540-92670 (so it's 530-92303 -> 540-92744 instead of 530-92303 -> 540-92670 -> 540-92744).

Which one is the case?

We are aware of the drawbacks when going from 7.7.x to 7.5, but this was the fastest solution as the CU 540-92744 didn't seem to solve the issue immediately.

[bbarmanroy]

Hi @RHolzer once a new CU is released, the older one is no longer available for being pushed out to endpoints. So you're right, it'd be option 2 as you've mapped out.

You can also tweak the delay values to a higher number.