Globalprotect vpn disconnects few times a day

[rgunna2020]

Hi Guys,

Users are logged out of vpn few times a day and reconnected back in Milliseconds. Below is the log which shows error 10052

log:

(T1636)Debug( 278): 02/01/21 07:54:52:256 IPSec tunnel receive failed with error 10052(The connection has been broken due to keep-alive activity detecting a failure while the operation was in progress.)
(T1636)Error(1357): 02/01/21 07:54:52:257 VPN: Socket Failed to receive! ret = -1
(T1636)Info (1944): 02/01/21 07:54:52:257 ProcPackets, RecvFromSocket() failed
(T1636)Info ( 567): 02/01/21 07:54:52:257 ProcPackets failed, get out of ProcMonitor
(T1636)Debug( 626): 02/01/21 07:54:52:257 Tunnel downtime is 78 miliseconds
(T1636)Debug(5579): 02/01/21 07:54:52:257 Show Gateway vpn.fsource.org: Checking network availability and restoring VPN connection when network is available.
(T1636)Debug(6645): 02/01/21 07:54:52:257 --Set state to Restoring VPN Connection
(T7676)Debug(2381): 02/01/21 07:54:52:257 Setting debug level to 5
(T1636)Debug( 662): 02/01/21 07:54:52:258 Stop ProcDrv before disconnect
(T1832)Info ( 951): 02/01/21 07:54:52:258 ProDrv: VPN disconnect event, get out of ProcDrv
(T1832)Info ( 980): 02/01/21 07:54:52:258 ProcDrv thread dies
(T1636)Info ( 914): 02/01/21 07:54:52:258 ProcDrv quit
(T1636)Debug( 667): 02/01/21 07:54:52:258 Stop ProcTunnel before disconnect
(T3592)Info (1031): 02/01/21 07:54:52:258 ProTunnel: VPN disconnect event, get out of ProcTunnel
(T3592)Info (1047): 02/01/21 07:54:52:258 ProcTunnel thread dies
(T1636)Info ( 995): 02/01/21 07:54:52:258 ProcTunnel quit
(T1636)Debug( 229): 02/01/21 07:54:52:259 IPSec anti-replay statistics: outside window count 0, replay count 0
(T1636)Debug( 231): 02/01/21 07:54:52:259 Disconnect udp socket
(T1636)Debug( 547): 02/01/21 07:54:52:259 unset network
(T1636)Debug(2722): 02/01/21 07:54:52:259 UnsetRoutes(): RestoreDefaultRoutes.
(T1636)Debug(2728): 02/01/21 07:54:52:259 Unset 3 routes
(T1636)Debug(2748): 02/01/21 07:54:52:259 UnsetRoutes: DeleteIpForwardEntry[0] (0.0.0.0)
(T1636)Debug(2748): 02/01/21 07:54:52:259 UnsetRoutes: DeleteIpForwardEntry[1] (10.1.1.16)
(T1636)Debug(2748): 02/01/21 07:54:52:259 UnsetRoutes: DeleteIpForwardEntry[2] (10.1.1.15)
(T1636)Debug(6452): 02/01/21 07:54:52:259 UnsetGatewayRoutes: DeleteIpForwardEntry(209.217.222.189)
(T1636)Info (4952): 02/01/21 07:54:52:259 RemoveGatewayInRouteTable(vnicIdx=21)
(T1636)Info (5000): 02/01/21 07:54:52:259 delete 1 ip forward entry: 10.22.2.192
(T1636)Info (5000): 02/01/21 07:54:52:259 delete 2 ip forward entry: 192.168.86.0
(T1636)Info (5000): 02/01/21 07:54:52:259 delete 3 ip forward entry: 192.168.86.255
(T1636)Info (5000): 02/01/21 07:54:52:259 delete 4 ip forward entry: 224.0.0.0
(T1636)Info (5000): 02/01/21 07:54:52:259 delete 5 ip forward entry: 255.255.255.255
(T1636)Debug(2687): 02/01/21 07:54:52:259 UnsetRoutesV6: No route installed before
(T1636)Debug(1408): 02/01/21 07:54:52:259 Disconnect virtual interface
(T13376)Debug(6197): 02/01/21 07:54:52:672 NetworkConnectionMonitorThread: route change detected. Wait for 3 seconds.
(T13376)Debug(5025): 02/01/21 07:54:52:672 No need to check gateway route since no tunnel.
(T1636)Debug( 25): 02/01/21 07:54:52:733 create thread 0x69c with thread ID 2880
(T1636)Debug(1424): 02/01/21 07:54:52:733 Start FlushDNSCache thread 0x69c
(T2880)Debug(2223): 02/01/21 07:54:52:740 FlushDNSCache thread: run cmd: cmd /C ipconfig /flushdns > "C:\Program Files\Palo Alto Networks\GlobalProtect\4"
(T9060)Debug( 530): 02/01/21 07:54:52:743 ST,exit reader set!!!!(T9060)Debug(3062): 02/01/21 07:54:52:743 ST,End gp filter reader
(T1636)Debug(2442): 02/01/21 07:54:52:831 ST,service status is 00000004
(T1636)Debug(2447): 02/01/21 07:54:52:831 ST,service already started
(T1636)Debug(2450): 02/01/21 07:54:52:886 ST,Service stop send, current status = 00000001.
(T1636)Debug(1195): 02/01/21 07:54:52:886 SPStop is called
(T1636)Debug( 848): 02/01/21 07:54:52:954 restorednssuffix: dnsSuffix:fsource.local
(T7676)Debug(2381): 02/01/21 07:54:52:970 Setting debug level to 5
(T13376)Debug(6258): 02/01/21 07:54:55:708 NetworkConnectionMonitorThread: m_state = 0, m_bOnDemand=1, m_bAgentEnabled=1, m_bJustResumed is 0,
m_bHibernate is 0, m_bAgentEnabled is 1, m_bDisconnect is 0, IsConnected() is 0, IsVPNInRetry() is 1.
(T13376)Debug(5025): 02/01/21 07:54:55:708 No need to check gateway route since no tunnel.
(T13376)Debug(6266): 02/01/21 07:54:55:708 Set retry network check event in retry mode
(T13376)Debug(6278): 02/01/21 07:54:55:708 NetworkConnectionMonitorThread: Detected route change, but skip network discovery.
(T1636)Error(5310): 02/01/21 07:54:56:358 DLSA, wait time out for monitor route change thread, time out = 3 seconds
(T1636)Error(1821): 02/01/21 07:54:56:358 DLSA, route changing monitor thread leaked
(T1636)Debug(5982): 02/01/21 07:54:56:358 DLSA, savedMetric1Routes not present, do not need to restore
(T1636)Debug(5347): 02/01/21 07:54:56:358 Proxy is not disabled before, no need to restore
(T1636)Debug( 839): 02/01/21 07:54:56:380 PreviousDNSInfo doesn't exist, no need to restore
(T1636)Debug(1874): 02/01/21 07:54:56:389 UnsetDNSSuffixSearchOrder returns 0
(T1636)Debug(10848): 02/01/21 07:54:56:392 SetVpnStatus called with new status=0, Previous Status=1
(T1636)Debug(4124): 02/01/21 07:54:56:392 UpdatePrelogonStateForSSO() - User-logon tunnel state = Disconnected

[BPry]

@rgunna2020,

From the logs you uploaded, the agent believes that the tunnel has failed due to keepalive traffic failing. Do you see this across all clients at the same time, or random clients throughout the day? This would really only be a concern if it's happening to multiple clients at the same exact time, otherwise it could easily be, and likely is, brief disruptions to the users network access. 

[rgunna2020]

@BPry @this happening to all the users at the same time

[BPry]

@rgunna2020 ,

Whenever you run into an issue like that the most immediate questions are these two:

1. Is the site the users connecting to experiencing any WAN connection issues during the same time period?
2. Do you have any sort of zone or DoS setup on your portal/gateway?

The first one is straightforward. Assuming that you have WAN monitoring already in place, do you see any service degradation at the time users are experiencing the issue? IPSec is susceptible to disruptions, so an issue on your WAN connection that you maybe wouldn't notice in the office can become a severe issue while working remotely.

The second issue that you would want to look at is if your system is logging any zone or DoS alerts during the same time period, or do you even have these features configured. With everyone working remotely I've seen a lot of instances where folks forget about re-analyzing these features and start hitting activate and maximum rates that they aren't actually alerting on.

That's the first two things that I would look at. If it's neither of those you can did into the sslvpn_php and sslvpn_ngx logs, but I would look at the two above first. 

[sebastianvd]

Some disconnect issues have been fixed:

A hotfix for 5.2.5 has been released with a lot of fixes btw: Addressed Issues in GlobalProtect App 5.2 (paloaltonetworks.com)

[KarolMajewski]

There are most likely intermittent network issues.
In my case there was a loop at the ISP network which lead to packets reaching their TTL value.

The tunnel drops happened around the same time when the client received an "ICMP time-to-live exceeded" packet.

I recommend looking on the wire (i.e. with wireshark) on the physical interface (not the virtual GP adapter). 

As a quick workaround you could block incoming ICMP TTL exceeded packets either on the router level or local firewall or increase the IP TTL value in the registry (Windows 10 defaults to 128) should the packet capture reveal that this is actually the case.