GlobalProtect immediate gateway-logout after gateway-register, no errors to be found in firewall monitoring

[Ranger-IT]

Hi LIVEcommunity,

starting yesterday a select few (but increasing) amount of our GlobalProtect users can't establish a connection anymore. I checked the firewall's logs and can't find any hint as to what causes the issues, as most other users seem to be able to connect and work just fine.

The GlobalProtect monitoring shows this:

Receive Time	Status	Stage	Event	Private IPv4	Auth-Method
02/04 10:22:34	success	logout	gateway-logout	0.0.0.0
02/04 10:22:34	success	login	gateway-register	0.0.0.0
02/04 10:22:34	success	login	gateway-auth	0.0.0.0	cookie
02/04 10:22:34	success	before-login	gateway-prelogin	0.0.0.0

Interestingly, the reported GlobalProtect app version is "0.0.2" for the users that have trouble - this must be a false reporting though, because we deploy only 5.2.8 and pre-install this on all machines as well - not to mention these same users were connected just shortly before.

The GlobalProtect gateway should have free available IPv4 addresses in its pool, and I assume if it didn't it would hopefully show an error? None of the certificates are expired either, we see no authentication errors, I'm unsure what could cause this.

Any ideas? I'll take a closer look at an affected client soon and will provide more info if possible.

Thanks in advance!

[Ranger-IT]

I have since done a repair-installation on 2 affected clients and they worked fine right afterwards without any further action.

I also found the following entries in the PanGPS.log:

(P6868-T10936)Debug(5761): 02/04/22 12:14:26:201 Created gateway route (1.2.3.4) succeeds
(P6868-T10936)Debug( 230): 02/04/22 12:14:26:201 set virtual interface driver started as yes
(P6868-T10936)Error( 246): 02/04/22 12:14:26:201 objdb.SetDoc() failed: 
errors getting SSL/VPN config
(P6868-T10936)Error( 857): 02/04/22 12:14:26:201 Failed to set client config
(P6868-T10936)Error(2512): 02/04/22 12:14:26:201 CreateTunnel: SetConfig() failed
(P6868-T10936)Debug(7351): 02/04/22 12:14:26:201 UnsetGatewayRoutes: DeleteIpForwardEntry(1.2.3.4)
(P6868-T10936)Debug(7002): 02/04/22 12:14:26:201 --Set state to Connection failed
(P6868-T10936)Debug(2528): 02/04/22 12:14:26:202 VPN tunnel is not connected.
(P6868-T10936)Debug(2530): 02/04/22 12:14:26:202 returns FALSE.
(P6868-T10936)Debug(2622): 02/04/22 12:14:26:202 failed to create tunnel with gateway Gateway
(P6868-T10936)Info (2295): 02/04/22 12:14:26:202 logout: user=User, portal=Portal, gateway=Gateway, domain=Domain, computerName=Hostname

PanGPA.log shows this at the same time:

(P8104-T10372)Debug( 121): 02/04/22 12:14:26:202 Received data from Pan Service
(P8104-T10372)Debug( 608): 02/04/22 12:14:26:202 Current status is changed to 1.
(P8104-T10372)Debug( 174): 02/04/22 12:14:26:202 username field is not empty. not override the username.
(P8104-T10372)Debug( 203): 02/04/22 12:14:26:202 CPanBaseReceiver::HandleStatus - found discover-ready tag. value = n.
(P8104-T10372)Debug( 210): 02/04/22 12:14:26:202 CPanBaseReceiver::HandleStatus - found cdl-log tag. value = n.
(P8104-T10372)Debug( 274): 02/04/22 12:14:26:202 message type from the service = s 
<?xml version="1.0" encoding="UTF-8"?>
<response>
	<type>status</type>
	<status>Disconnected</status>
	<protocol/>
	<portal-config-version>4100</portal-config-version>
	<error-must-show/>
	<error-must-show-level>error</error-must-show-level>
	<error>The network connection is unreachable or the gateway is unresponsive. Check the network connection and reconnect.</error>
	<product-version/>
	<product-code/>
	<portal-status>Connected</portal-status>
	<user-name>User</user-name>
	<username-type>regular</username-type>
	<state>Connection failed</state>
	<check-version>no</check-version>
	<portal>Portal</portal>
	<discover-ready>no</discover-ready>
	<mdm-is-enabled>no</mdm-is-enabled>
	<gateway-list name="gateway-list" type="external" user="User">
		<no-gateway>true</no-gateway>
		<entry>
			<gateway>Gateway</gateway>
			<tunnel>no</tunnel>
			<description>Gateway</description>
			<allow-tunnel>yes</allow-tunnel>
			<passwd-expire-days>-1</passwd-expire-days>
			<priority>1</priority>
			<internal>no</internal>
			<authenticated>yes</authenticated>
		</entry>
	</gateway-list>
	<cdl-log>no</cdl-log>
</response>

It so far only affects 3 users, of which one is a new machine and 2 were existing machines that previously connected to GlobalProtect just fine. I have only one potential cause for why this happened to these three:

We use pre-vpn-connect.bat and post-vpn-connect scripts.bat scripts on our machines. We used to simply run a install.bat-script to distribute these two files and the required registry keys remotely to machines that needed it. We never bothered to change this, it worked fine for years.

Recently, we switched the install.bat-script to distribute the scripts and registry keys to a PowerShell script. I tested the functionality on the 3 affected clients and it seemingly worked fine. The files are identical and the registry keys are identical to when we used the install.bat. However, it seems it somehow still broke GlobalProtect?

Funny enough, a GlobalProtect "repair installation" fixed the issues and the pre-vpn-connect.bat and post-vpn-connect.bat scripts run fine as well, not to mention the PanGPS.log shows them running successfully, so I really don't believe they could be the cause of this problem, but I'll make sure to investigate. Maybe copying the two files into the "C:\Program Files\Palo Alto Networks"-folder remotely using PowerShell somehow breaks that folder or its access rights or so?

Just figured I'd post the "solution" here in case anyone experiences this as well. I'll try to confirm if the PowerShell script, despite doing nothing but copying 2 .bat-files and the 2 registry keys to the machine, is the reason for this issue.

[Ranger-IT · Accepted solution]

Update:

The PowerShell script was faulty. It was overwriting the already existing registry key "HKLM\SOFTWARE\Palo Alto Networks" completely, because unless you specify the parameter "-force", you have to create each sub-key one by one. This was fixed by simply creating the "post-vpn-connect" and "pre-vpn-connect" keys with parameter "-force".

Long story short: I was dumb. 🙂