Global Protect Client disconnect Issues after upgrading to Pan-OS 11.1.0

[UtkarshKumar]

Hi Team

We upgraded Palo Alto FW to 11.1 and started having VPN Global Protect Client Issues where it would disconnect/reconnect multiple times.

It would get connected and there wouldn't be any internet access, wouldn't allow any traffic so something isn't working. Flipped back over to the passive firewall which hadn't been upgraded yet and everything stabilized.

Seems to be some issue with the 11.1.0 and GP Client versions:6.21. 6.22 and 6.20 clients. We tried connecting with all of them

In PanGPS.log

We see multiple disconnects:

(P4084-T2576)Debug(6677): 11/30/23 13:08:42:870 NetworkDiscoverThread: PortalStatus is 2, HasLoggedOnGateway is 1
(P4084-T2576)Debug(6810): 11/30/23 13:08:42:870 Reset NetworkDiscovery waitTime to 5 seconds.
(P4084-T2576)Debug(6151): 11/30/23 13:08:42:870 NetworkDiscoverThread: wait for network discover event.
(P4084-T15224)Debug(5374): 11/30/23 13:08:42:871 Found virtual IP route entry
(P4084-T15224)Debug(6137): 11/30/23 13:08:42:871 No change for gateway route
(P4084-T16620)Debug(12635): 11/30/23 13:08:42:871 m_preUsername cdougall
(P4084-T16620)Debug(1449): 11/30/23 13:08:42:871 m_msp->IsVPNConnected() is 1, CControlManager::GetInstance()->IsInRetry() is 0
(P4084-T16620)Debug(7692): 11/30/23 13:08:42:872 --Set state to Disconnecting...
(P4084-T16620)Debug(1499): 11/30/23 13:08:42:872 AddAttribute for proxy agent
(P4084-T16620)Info (2744): 11/30/23 13:08:42:873 Disconnect(VPN recv failed) called
(P4084-T1664)Debug(5002): 11/30/23 13:08:42:873 LifeTimeThread receives m_hExitLifeTimeThreadEvent
(P4084-T1664)Debug(5030): 11/30/23 13:08:42:873 LifeTimeThread quits
(P4084-T16620)Debug(1132): 11/30/23 13:08:42:873 vpn disconnect

We see this repeating every 30-40 seconds:

(P4084-T17104)Debug(6677): 11/30/23 13:13:05:869 NetworkDiscoverThread: PortalStatus is 1, HasLoggedOnGateway is 1
(P4084-T17104)Debug(6810): 11/30/23 13:13:05:869 Reset NetworkDiscovery waitTime to 5 seconds.
(P4084-T17104)Debug(6151): 11/30/23 13:13:05:869 NetworkDiscoverThread: wait for network discover event.
(P4084-T17312)Debug(5374): 11/30/23 13:13:05:870 Found virtual IP route entry
(P4084-T17312)Debug(6137): 11/30/23 13:13:05:870 No change for gateway route
(P4084-T17280)Debug(12635): 11/30/23 13:13:05:870 m_preUsername cdougall
(P4084-T17280)Debug(1449): 11/30/23 13:13:05:870 m_msp->IsVPNConnected() is 1, CControlManager::GetInstance()->IsInRetry() is 0
(P4084-T17280)Debug(7692): 11/30/23 13:13:05:871 --Set state to Disconnecting...
(P4084-T17280)Debug(1499): 11/30/23 13:13:05:871 AddAttribute for proxy agent
(P4084-T17280)Info (2744): 11/30/23 13:13:05:871 Disconnect(VPN recv failed) called
(P4084-T8188)Debug(5002): 11/30/23 13:13:05:871 LifeTimeThread receives m_hExitLifeTimeThreadEvent
(P4084-T8188)Debug(5030): 11/30/23 13:13:05:871 LifeTimeThread quits
(P4084-T17280)Debug(1132): 11/30/23 13:13:05:871 vpn disconnect
(P4084-T17280)Debug(1133): 11/30/23 13:13:05:871 Delete m_vpn in CControlManager::DisconnectVPN()

We have checked and there is no known issue reported or at least documented in the Palo Publick link. Anyone faced this similar issue?

Thanks and Regards

GlobalProtect 

[TTNhan]

Are there any suggestions from experts on this issue?

I have many customers with the same issue.

issue found on 11.0.3 and 11.1.0

[TTNhan]

I have collect the PAN-GPS.log

----log----

P5076-T10724)Debug(5119): 12/19/23 09:18:29:556 Wait for virtual interface IP route entry 172.16.254.16
(P5076-T10724)Dump (5120): 12/19/23 09:18:29:556 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->dwNumEntries is 15
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:556 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:556 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:556 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:619 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:619 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:619 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:682 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:682 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:682 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:743 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:743 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:743 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:807 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:807 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:807 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:870 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:870 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:870 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:932 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:932 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:932 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:994 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:29:994 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:29:994 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:057 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:057 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:057 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:119 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:119 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:119 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:182 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:182 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:182 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:242 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:242 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:242 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:307 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:307 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:307 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:370 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:370 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:370 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:431 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:431 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:431 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:494 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:494 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:494 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:554 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:554 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:554 Didn't find the route entry. Try again.
(P5076-T5276)Dump ( 882): 12/19/23 09:18:30:617 HandleDnsCallBack enter...
(P5076-T5276)Dump ( 927): 12/19/23 09:18:30:617 HandleDnsCallBack isPv6=0, from virtual interace=0
(P5076-T5276)Dump ( 936): 12/19/23 09:18:30:617 HandleDnsCallback: dns request..
(P5076-T5276)Dump ( 943): 12/19/23 09:18:30:617 HandleDnsCallback: initReplyBuffer done.
(P5076-T5276)Dump ( 91): 12/19/23 09:18:30:617 Received DNS request for geo.prod.do.dsp.mp.microsoft.com with type 1
(P5076-T5276)Dump ( 441): 12/19/23 09:18:30:617 EnforceSplitDns, no tunnel yet, let dns qry passthrough.
(P5076-T5276)Dump ( 953): 12/19/23 09:18:30:617 HandleDnsCallback result=passthrough
(P5076-T5276)Dump ( 784): 12/19/23 09:18:30:617 ST,lasterror is 997
(P5076-T5276)Dump ( 786): 12/19/23 09:18:30:617 ST,lasterror is ERROR_IO_PENDING
(P5076-T5276)Dump ( 789): 12/19/23 09:18:30:617 ST,write success
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:617 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:617 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:617 Didn't find the route entry. Try again.
(P5076-T5276)Dump ( 882): 12/19/23 09:18:30:632 HandleDnsCallBack enter...
(P5076-T5276)Dump ( 927): 12/19/23 09:18:30:632 HandleDnsCallBack isPv6=0, from virtual interace=0
(P5076-T5276)Dump ( 930): 12/19/23 09:18:30:632 HandleDnsCallback: dns response..
(P5076-T5276)Dump ( 91): 12/19/23 09:18:30:632 Received DNS request for geo.prod.do.dsp.mp.microsoft.com with type 1
(P5076-T5276)Dump ( 932): 12/19/23 09:18:30:632 handleDnsCallback: dnsRes parse returns:0
(P5076-T5276)Dump ( 953): 12/19/23 09:18:30:632 HandleDnsCallback result=passthrough
(P5076-T5276)Dump ( 784): 12/19/23 09:18:30:632 ST,lasterror is 997
(P5076-T5276)Dump ( 786): 12/19/23 09:18:30:632 ST,lasterror is ERROR_IO_PENDING
(P5076-T5276)Dump ( 789): 12/19/23 09:18:30:632 ST,write success
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:680 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:680 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:680 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:744 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:744 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:744 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:806 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:806 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:806 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:869 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:869 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:869 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:930 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:930 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:930 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:993 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:30:993 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:30:993 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:055 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:055 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:31:055 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:118 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:118 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:31:118 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:180 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:180 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:31:180 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:242 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:242 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:31:242 Didn't find the route entry. Try again.
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:304 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[1].dwForwardNextHop is 127.0.0.1
(P5076-T10724)Dump (5129): 12/19/23 09:18:31:304 CPanNetSetup::GetRouteTableEntry(): pIpForwardTable->table[11].dwForwardNextHop is 0.0.0.0
(P5076-T10724)Dump (5156): 12/19/23 09:18:31:304 Didn't find the route entry. Try again.
(P5076-T5276)Dump ( 882): 12/19/23 09:18:31:336 HandleDnsCallBack enter...

-------endlog----------

[lestrada]

I am having the same issue Upgraded from 9.0.14 to 9.1.17 on Dec 27, 2023

Has your issue being resolved as of Jan 2, 2024?

[TTNhan]

I  also have a customer report this issue when upgrading from 9.1.16 to 9.1.17.

They have to rollback for workaround

[RudolfTorvenyi1]

my customer is facing the same issue after upgrade from 11.0.2-h2 -> 11.0.3

another customer had the same issue after upgrade from 10.2.7 -> 10.2.7-h3. After downgrade back to 10.2.7 GlobalProtect VPN is stable.

[lestrada]

I had to roll back as well- Issue solved

[Architectural]

Is downgrading the version straightforward? Can someone list the steps or point to documentation?

Do we know which version(s) to avoid which have these GlobalProtect issues?  Major/minor v11, 10 & 9

[TTNhan]

I believe many customers faced the same issue but could not open the case because they had to roll back right away and could not keep the firewall at PAN-OS from having a bug.

I expect the backend Palo Alto  team should do a test  at 9.1.17, 10.1.11, 10.2.7, 11.0.3, 11.1.0

[MP18]

@TTNhan I agree with you i also have issues while doing PAN OS upgrade.

I have no issues with current PAN OS 10.0.11-h3 as i also need to roll back to make sure GP is working

Regards

[UtkarshKumar]

We also had to rollback to resolve the issue with GP disconnects.

[Jeff_Starling]

Is anyone from Palo Alto Support looking at this issue?  Considering the number of versions that seem to be reported, this is not a trivial issue.

[GlobetrotterCT]

We are having same issue after upgraded from PanOS 10.1.11-h1 to 10.1.11-4

Global Protect Client 6.2.2

Doesnt seem to impact everyone but the people it does impact weve managed to workaround by completely removing GlobalProtect VPN client, wiping config from their local appdata then reinstalling and it seems to resolve for a while.

Palo Alto seem to be a bit like Microsoft support - you have to wait till enough people complain about something before it gets any attention.

[emr_1 · Accepted solution]

As far as I know, I got update from TAC as below:

===

+Issue ID:
...PAN-242561

+Workaround:
...disable IPv6 on GP Virtual Network Adapter (on client machine side)

OR

...use IPSec mode. The issue is seen by SSL mode at this point.

+Fix Schedule:

...target release for fix (current latest release FYI)
...11.2.0 (not released yet)
...11.1.2 <** updated on 27th/Feb/2024 **>
...11.0.5 (11.0.3-h3)
...10.2.8 (10.2.7-h3)
...10.1.14 (10.1.12)

===

[GlobetrotterCT]

I can confirm in my case that disabling IPV6 on the PANGP Virtual Ethernet Adapter then restarting the PanGPS service and reconnecting  appears to resolve the issue.