Troubleshooting random GlobalProtect disconnects...

[jrauman]

Hello. i have been experiencing random GlobalProtect disconnects on my home computer.  I'm running Windows 10 [1909] with GlobalProtect 5.0.8 64-bit connecting back to my office's Palo Alto firewall (not 100% sure of the version). A few times a day, GlobalProtect will just disconnect on its own. I am typically running a Remote Desktop Connection from my home PC to my work PC when this happens and the RD session will freeze and disconnect as well.  I started collecting logs when this happens but dont really know how to parse them.  I have noticed a few things in various logs.   

• In the  pan_gp_event log, i see an Info message "Tunnel is down due to network change."
• In the PanGPA log, i see the Info message "Window session changed with state 7", followed by several Debug log entries.
  • Debug(2825): user session locked.
  • Debug( 669): balltip, m_hipMsg=00000171893379A0
  • Debug( 672): balltip, getcount return 0
  • Debug( 766): CAC, type is 8004, data=0000006C7E4FF7C0
  • Debug( 810): CAC, DBT_DEVICEREMOVECOMPLETE, device type=00000005, cacUnplugLogout=0
  • Debug( 851): CAC, do nothing for device remove message
  • (NOTE: Those last three repeat a dozen or so times).
  • Debug( 93): Received data from Pan Service
  • Debug( 172): username field is not empty. not override the username.
  • Debug( 193): CPanBaseReceiver::HandleStatus - found discover-ready tag. value = y.
  • Then i get a message that says "Gateway <portal address>: Checking network availability and restoring VPN connection when network is available.
  • Then some messages about trying to restore the connection
• In the PanGPS log i see a Debug message "Received session change, event type 7, session 1" followed by and Info message "lock off  session 1"

How would you recommend troubleshooting these disconnects.  I'm not familiar enough with the firewall side of things to know how to look at any logging on that side.  Is the issue with my home computer/OS/hardware/network?  or is it likely something in GlobalProtect and/or Palo Alto that needs attention?

Thank You!

[Mick_Ball]

-Firstly I would go for the info in the PanGPS log on the local device.and perhaps paste more info.

also check your PC event viewer - Applications and Service Logs->Microsoft->Windows->Wlan-Autoconfig. operational to see if wifi is playing up.

the palo alto system logs @ Monitor/system  may suggest why this happened.  i normally see a client disconnect message but at least you then know it's not a firewall issue.

[MichaelBredell]

Did you find a solution to this? I am experiencing the same problem.

[jrauman]

Thank you for the suggestions.  Soon after i entered the request, i got stuck at work for the last three weeks and couldn't use GlobalProtect from home at all so i never got around to trying your suggestions.  I'm finally home for the weekend and will look for those logs you mentioned and will paste them here after my next disconnect.  

[jrauman]

I didnt find a solution yet. but now my whole company is using this product so i better find one really really soon. Hope to get some use out of it this weekend.  I'll post back as soon as i have new info/

[DanMarchewka]

Did you ever find a solution to this issue?  We are having the same issue.  GP works great, but RDP sessions to internal PCs freeze, causing you to have to close the session and RDP back into the machine.  Sometimes it happens once a day, sometimes it happens several times an hour.  We are running 9.0.7 and GP 5.1.1.  We did not have this happen on our old SonicWall SMA410 and NetExtender.  It's was definitely introduced along with our Palo deployment 2 weeks ago.

If anyone has a solution to this issue, can you please post it?  

Thanks!

[Bharathidasan]

Hi 

I was getting frequent disconnect issues in GP 5.1.1 version, please confirm whether 5.1.3 version is recommended to upgrade, cause this version also has a lot of known issues.\

GPC-10574	Fixed an issue where, when the GlobalProtect app was installed on Windows with a different language other than English (for example, Spanish), the GlobalProtect agent was continuously restarted.
GPC-10535	Fixed an issue where, after you upgraded the GlobalProtect app from 5.0.x release to 5.1.1 release on a Mac device, users were prompted to re-enter their password even when the saved password was set to  Yes . With this fix, users will not be prompted to re-enter their password after the upgrade. For GlobalProtect to access user credentials from the login keychain, the following Keychain Pop-Up message will appear: GlobalProtect wants to use your confidential information stored in "GlobalProtect" in your keychain. Users are prompted to enter their password and then select  Always Allow  so that the Keychain Pop-Up prompt does not appear again.
GPC-10468	Fixed an issue where, when the GlobalProtect app was installed on Windows, two OpenSSL DLL files in 64-bit were not signed by a Palo Alto Networks certificate. This issue caused a problem for some endpoint protection applications.
GPC-10403	Fixed an issue where the GlobalProtect app for macOS was disabled and the  Disable Timeout (min)  value expired, GlobalProtect could reconnect and user credentials were not preserved.
GPC-10395	Fixed an issue where the GlobalProtect app for macOS version 5.1.1 could not be properly installed because the GlobalProtect service failed to launch.
GPC-10380	Fixed an issue where the GlobalProtect app on macOS displayed the following error message when all the gateways were configured as  Manual Only  priority: Could not connect to Gateway, Contact your IT administrator With this fix, the app now displays the following message: Please select a gateway to connect manually
GPC-10341	Fixed an issue on Windows endpoints where, after the endpoint woke up from sleep mode, the GlobalProtect app was disconnected and then attempted to reconnect to the portal or gateway.
GPC-10311	Fixed an issue where, when the GlobalProtect app was installed on macOS and Windows, cookie authentication was successful even when the wrong password was used and GlobalProtect was still connected after users sign out of the app. With this fix, authentication cookies are now deleted from the system when users sign out of the app.
GPC-10288	Fixed an issue where, when GlobalProtect was installed using the Windows Installer (Msiexec) with on-demand as the connect method, GlobalProtect automatically tried to connect to the portal.
GPC-10261	Fixed an issue where the GlobalProtect app displayed the customized  Captive Portal Detection Message  in the wrong format when a different language was used other than English.
GPC-10227	Fixed a connectivity issue where, when the GlobalProtect app was installed for macOS Catalina, the GlobalProtect connection was periodically lost.
GPC-10228	Fixed an issue where the GlobalProtect app detected the presence of a captive portal even though it was not present.
GPC-10118	Fixed a periodic issue where the GlobalProtect tunnel failed to be restored after waking up from sleep mode. This issue occurred when on-demand was used as the connect method.
GPC-10024	Fixed an issue where, after upgrading to GlobalProtect 5.0.6, the GlobalProtect HIP check did not detect that Symantec Endpoint Protection 14.2 real-time protection was enabled, which caused the device to fail the HIP check.
GPC-10190	Fixed an issue where the GlobalProtect app on macOS failed to find the correct certificate for authentication to the gateway, when the object identifier (OID) was specified in the plist. With this fix, when you provide the Key Usage OID in the plist, the GlobalProtect app uses the correct certificate.
GPC-9913	Fixed an issue where the portal configuration selection criteria failed when the certificate was signed with the version 2 template.
GPC-9779	Fixed an issue that caused the GlobalProtect app to install a default route with the same metric as the system default route when split-tunneling based on access route and destination domain was enabled. This issue caused some excluded traffic to go through the tunnel.
GPC-9730	Fixed an issue where GlobalProtect failed to connect to the external gateway when the proxy was not reachable outside of the corporate network until the GlobalProtect service or the desktop was restarted.
GPC-9500	Fixed an issue in GlobalProtect for macOS endpoints where installing or upgrading the package using a Mobile Device Management (MDM) solution such as JAMF Pro resulted in a GlobalProtect app initialization failure.

[devavratoka]

This exact thing is happening to us. What's weird is we switched from GP on prem to Prisma and it started with Prisma. We disabled UDP on the RDP client on everyone's PC in the reg key settings and it seemed to reduce the number of disconnects, but they're still happening.

[DanMarchewka]

It's very frustrating.  It started happening to us as soon as we put the Palo in 2+ weeks ago.  We are running PanOS 9.0.6 and GP 5.1.1.

What versions are you running?

[devavratoka]

Yeah! And TAC can’t see anything unusual either. 
we’re on prisma and GP 5.0.9-15

[DanMarchewka]

Yeah, my ticket with TAC has been open for a week now and it's been crickets.  If I hear something I will post on here.  Can you please do the same?

[devavratoka]

Absolutely. Btw, try disabling UDP on the RDP client and see if it helps you. It reduced the number of disconnects for us. 

check out https://getadmx.com/?Category=Windows_10_2016&Policy=Microsoft.Policies.TerminalServer::TS_CLIENT_TU...

[DanMarchewka]

We can look into it, but it's only happening on our BYOD policy using personal PCs.  Our company laptops aren't affected since they aren't RDP'ing into a work PC, just connecting to the network.  The issue when using a personal device to connect via GP and then RDP to their work PC at the office.  That is when we get the freezing and have to disconnect and reconnect.  Since these are personal devices, it would be hard to get everyone to configure turning UDP for RDP to disabled.

[jrauman]

Hi. Original Poster here. We have made some progress on these issues. It appears there were two separate issues at play.

1. Remote Desktop Connection freezing up several times a day (up to 8-12 times for some users).

2. GlobalProtect disconnecting suddenly (which causes RDC to disconnect, not freeze)..

For #1, we dont think it had anything to do with GlobalProtect.  We found that users who had Wi-Fi connections were the ones with the RD freezing up. Our resolution included ensuring the user was on 5GHz if available, and that their 5G wi-fi network didnt have any interference from neighboring wireless networks (we used Wi-Fi Analyzer from the Microsoft Store to find any).  If there was another 5G WiFi network on the same channel as the user's network, we asked them to try to move their WiFi network to another channel that wasnt being used by anyone else nearby.  This worked really well for many users.  We did find one exception. For users who have Roku, Chromecast, Amazon FireTV type devices, those devices will create a separate WiFI network on the same channel as the WiFi network itself (i dont know why). If someone else uses those devices, RDC will freeze up.  I would have users unplug those devices during working hours with good success. I reached out to Roku but they said "nothing we can do about it" and didnt explain why it was ncessary to throw up a separate network on the same channel and not at least give users the option to move the Roku network to another channel. Oh well.  (NOTE: we also bought some really long ethernet cables for users with this scenario if they couldnt do anything about Wi-FI interferrence.  That helped tremendously).

2. For the GP disconnects, one thing we found was that GP was set to the default timeout of 8 hours.  So every day around 3-5pm, users GP would disconnect (which was before they were done workign for the day).  We changed that to 12 and that issue went away.  We are now seeing very few GP disconnects at other times.  I'm going to do some testing this weekend on my machine to see if GP will disconnect over Ethernet over a period of a day.  I'll keep you posted.

Oh, someone mentioned disabling UDP for RDC.  We never had to do that to get great performance from RDC.  See #1 for our resolution.  Also we are trying to get Softphones to work over RDC and SIP traffic uses UDP if i recall correctly so that wouldnt work for us anyway.

[devavratoka]

@jrauman  

thanks. Our problems with RDP began after users switched from GP on prem to Prisma Access. Never had disconnects on their home network before.