How can I do a staggered roll out of Global Protect

[ksauer507]

I'm under the impression that if we change the version on our global protect portal, its all or nothing, and every client will attempt to upgrade from the firewall.

We've done this in the past and there's always about a 20% failure rate where we have to use remote support software and painstakingly manually fix people.  We want to avoid that.  Perhaps upgrade global protect to a department at a time.

Looking at version 6.1.4-C720.

We tried intune in the past but it cased issues.  One of the things with intune is you never know when its going to run.

[jonathanb]

I approach it using agent configuration and targeting by user.

I already have a group of users that 1st wave and another that are 2nd wave.

I set up the agent configuration to target 1st wave users and configure the update policy.

I set up the agent configuration to target 2nd wave users below 1st wave, and configure the update policy.

I set up the default agent configuration below that, with updates disabled.

Configure your updates as you'd like, either as a prompted update or a forced update.

When you've gone through the 1st wave and 2nd wave, update the default agent config to update the agent.

[ksauer507]

Ok this is an interesting approach, appreciate the feedback.

We have another set of Palos for a DR site.  Just testing 6.1.4 there vs 6.1.2 at the main site is problematic for automatic upgrades.

The upgrade seems to remove the ext-key-usage-oid-for-client-cert registry key in the Global Protect \ Settings HLKM area of the registry.  Then the user wouldnt even know they had to click on the globe icon becuase by default Windows hides the tray icon.  If you unhide tray icons and click it, theres at least 3 certificate to choose from.  The middle one is the OID that is defined in ext-key-usage-oid-for-client-cert registry key, so clicking that will connect you.  We want a seemless process.

We tried making the GlobalProtect\Settings subkey READ ONLY for system so the upgrade installer couldn't overwrite. We gave only Domain Admins R/W.  But then the upgrade installer errors out and the client is left with nothing at all.

I think we may have to open a case.  It doesn't seem like this is doable without manually installing with the EXTCERTOID=1.3.6.1.4.1.311.21.8.6716362.16708851.13517456.16705214.7242108.122.9016727.10419490 CERTIFICATESTORELOOKUP=machine MSI arguments.

[jonathanb]

I also have some registry objects that aren't handled well, like Connect Before Login settings.  I've decided to use an Intune remediation script to deal with it, but would have used AD group policy in the past.  I've also decided to use PSADT to wrap my initial installation and apply the reg objects after install.  You may need to install/upgrade using you RMM instead of the upgrade function built into the agent/firewall.

The installer does have a post-install script option.  I don't know if that re-runs during an upgrade.

The real problem is that there are settings for the agent that aren't available in the agent configuration, such as the ones we're both talking about.  PAN should fix that.

[ksauer507]

The problem with Intune (tried that in the past), is you never know when it runs.  Its so random that a user could be in the middle of working on something and it feels like nows the best time to install Global Protect, which uninstalls the old one to reinstall the new one, thus interrupting the end users connectivity and potentially disrupting their work.

We've tested the built in upgrade when a GP version is specified on a particular portal and the system waits exactly 2 minutes 30 seconds and then kicks off.  What if the user has slow internet and the GP client wasn't downloaded in time?  We think thats why in the past this method resulted in 10-20% failure rates, where GP would uninstall but then not have the new version to install - thus leaving them off the network.

Were working on writing a script that copies everything locally to a temp folder, then executes and installs it while presenting a dialog box to the end user each step of the process.  This way we can control the install and pass the proper msi installation variables.  A lot of the stuff is pushed via GPO and thats great but the upgrade wipes some of this out and we cant ask users to choose a cert.  Any of them seem to work but its not evident to the end user.  First the GP icon is hidden in the notification area in Windows.  You have to unhide them then when you see the flashing globe icon, click it and see 3 domain certificates.  Nothing works unitl one of these is clicked on.  Nothing pops up otherwise leading you to an infinitely disconnected situation.  I guess the UX designer was let go.

The big issue here is upgrades don't seem to keep the settings which is really bad.  Even if pushed via GPO or REG file import, it still is removed on upgrade.  This should be a bug and I think we might open a case but our script we are working on will probably be our solution.

[Adrian_Jensen]

On my GlobalProtect Portals I have done a few different things to manage GP client updates, though as you say, some clients always seem to fail one or more times before updating successfully. It looks like you are trying to push the update with scripting, but is your Portal set to auto-update or manually update the client?

Under the Portals settings the client config can be set to multiple options, at least 2 of which will try to force the client to update to the same version as the Portal is running (Network - GlobalProtect -> Portals -> [portal config] -> Agent -> [agent config] -> App -> Allow User to Upgrade GlobalProtect App). If your manually installing via Intune/etc. you should probably disable upgrades via the Portal client config.

It also depends on how your GP clients authenticate. By default, clients will attempt to reconnect to the Portal every 24 hours and download the Agent config again. If the PaloAlto GP client version has since changed, the client will download the new version. As the client needs to authenticate before it can download the new Agent config and update package, if you rely on a logged-in user or MFA, it may fail.

On my setup I have taken a different tact. My clients authenticate to the Portal with a machine/user certificate. This allows the client to always get the Agent config and update packages, even though the user may be away or the user creds have expired (my Gateway auth uses separate user SAML/MFA/etc.). I have the GP Upgrade set to "Allow Transparently". A few days before I am ready to update all the clients to a new version, I change the GlobalProtect Agent Config Refresh Interval down to 4-6 hours. Over the proceeding days all clients should change to checking for new configs every few hours. On the evening of the update I tell all users to leave their PCs connected and change the GP client version on the PA. Overnight, as the clients check in to verify the Agent config, they will see/download/install the new version and reconnect to the Portal. If one fails it reconnects, sees the update, and tries again. That takes care of about ~98% of the clients with a handful that need manual intervention. After the upgrades I roll the GP Agent Config Refresh back to 24 hours.

[ksauer507]

Ok we just built a signed powershell script that runs as part of the login scripts.  Login scripts work because we are an always on GP (pre-logon) shop.  It copies everything it needs to C:\Temp\GPUpgradeScript and executes the signed .ps1 file from that location against the msi in that same location.

First the script checks the user and if pangps is in existence.  Then it checks if version is less than 6.1.4-720, if it is it will install it with the arguments.

We can target this script on a user or user OU level so we have more control than our 6 figure Cadillac Palo Alto.

It took one of my guys 10 minutes to write that script.  Another way is a line to backup the reg settings and reimport them after the upgrade as part of the msi installer process.  Whats Palo's excuse?

[jonathanb]

A wrapper like PSADT can help avoid unexpected installs by allowing people to defer through the wrapper's UI.  a cheap workaround is to deploy the app as Available, send notices to people for a few days, then set it as required.