topic Re: Troubleshooting random GlobalProtect disconnects... in General Topics

Troubleshooting random GlobalProtect disconnects...

jrauman — Thu, 13 Feb 2020 10:54:00 GMT

Hello. i have been experiencing random GlobalProtect disconnects on my home computer. I'm running Windows 10 [1909] with GlobalProtect 5.0.8 64-bit connecting back to my office's Palo Alto firewall (not 100% sure of the version). A few times a day, GlobalProtect will just disconnect on its own. I am typically running a Remote Desktop Connection from my home PC to my work PC when this happens and the RD session will freeze and disconnect as well. I started collecting logs when this happens but dont really know how to parse them. I have noticed a few things in various logs.

In the pan_gp_event log, i see an Info message "Tunnel is down due to network change."
In the PanGPA log, i see the Info message "Window session changed with state 7", followed by several Debug log entries.
- Debug(2825): user session locked.
- Debug( 669): balltip, m_hipMsg=00000171893379A0
- Debug( 672): balltip, getcount return 0
- Debug( 766): CAC, type is 8004, data=0000006C7E4FF7C0
- Debug( 810): CAC, DBT_DEVICEREMOVECOMPLETE, device type=00000005, cacUnplugLogout=0
- Debug( 851): CAC, do nothing for device remove message
- (NOTE: Those last three repeat a dozen or so times).
- Debug( 93): Received data from Pan Service
- Debug( 172): username field is not empty. not override the username.
- Debug( 193): CPanBaseReceiver::HandleStatus - found discover-ready tag. value = y.
- Then i get a message that says "Gateway <portal address>: Checking network availability and restoring VPN connection when network is available.
- Then some messages about trying to restore the connection
In the PanGPS log i see a Debug message "Received session change, event type 7, session 1" followed by and Info message "lock off session 1"

How would you recommend troubleshooting these disconnects. I'm not familiar enough with the firewall side of things to know how to look at any logging on that side. Is the issue with my home computer/OS/hardware/network? or is it likely something in GlobalProtect and/or Palo Alto that needs attention?

Thank You!

Re: Troubleshooting random GlobalProtect disconnects...

Mick_Ball — Thu, 13 Feb 2020 16:19:44 GMT

-Firstly I would go for the info in the PanGPS log on the local device.and perhaps paste more info.

also check your PC event viewer - Applications and Service Logs->Microsoft->Windows->Wlan-Autoconfig. operational to see if wifi is playing up.

the palo alto system logs @ Monitor/system may suggest why this happened. i normally see a client disconnect message but at least you then know it's not a firewall issue.

Re: Troubleshooting random GlobalProtect disconnects...

MichaelBredell — Wed, 25 Mar 2020 11:10:52 GMT

Did you find a solution to this? I am experiencing the same problem.

Re: Troubleshooting random GlobalProtect disconnects...

jrauman — Sat, 28 Mar 2020 00:17:39 GMT

Thank you for the suggestions. Soon after i entered the request, i got stuck at work for the last three weeks and couldn't use GlobalProtect from home at all so i never got around to trying your suggestions. I'm finally home for the weekend and will look for those logs you mentioned and will paste them here after my next disconnect.

Re: Troubleshooting random GlobalProtect disconnects...

jrauman — Sat, 28 Mar 2020 00:19:17 GMT

I didnt find a solution yet. but now my whole company is using this product so i better find one really really soon. Hope to get some use out of it this weekend. I'll post back as soon as i have new info/

Re: Troubleshooting random GlobalProtect disconnects...

DanMarchewka — Thu, 14 May 2020 19:26:51 GMT

Did you ever find a solution to this issue? We are having the same issue. GP works great, but RDP sessions to internal PCs freeze, causing you to have to close the session and RDP back into the machine. Sometimes it happens once a day, sometimes it happens several times an hour. We are running 9.0.7 and GP 5.1.1. We did not have this happen on our old SonicWall SMA410 and NetExtender. It's was definitely introduced along with our Palo deployment 2 weeks ago.

If anyone has a solution to this issue, can you please post it?

Thanks!

Re: Troubleshooting random GlobalProtect disconnects...

Bharathidasan — Fri, 15 May 2020 10:49:36 GMT

I was getting frequent disconnect issues in GP 5.1.1 version, please confirm whether 5.1.3 version is recommended to upgrade, cause this version also has a lot of known issues.\

GPC-10574	Fixed an issue where, when the GlobalProtect app was installed on Windows with a different language other than English (for example, Spanish), the GlobalProtect agent was continuously restarted.
GPC-10535	Fixed an issue where, after you upgraded the GlobalProtect app from 5.0.x release to 5.1.1 release on a Mac device, users were prompted to re-enter their password even when the saved password was set to Yes . With this fix, users will not be prompted to re-enter their password after the upgrade. For GlobalProtect to access user credentials from the login keychain, the following Keychain Pop-Up message will appear: GlobalProtect wants to use your confidential information stored in "GlobalProtect" in your keychain. Users are prompted to enter their password and then select Always Allow so that the Keychain Pop-Up prompt does not appear again.
GPC-10468	Fixed an issue where, when the GlobalProtect app was installed on Windows, two OpenSSL DLL files in 64-bit were not signed by a Palo Alto Networks certificate. This issue caused a problem for some endpoint protection applications.
GPC-10403	Fixed an issue where the GlobalProtect app for macOS was disabled and the Disable Timeout (min) value expired, GlobalProtect could reconnect and user credentials were not preserved.
GPC-10395	Fixed an issue where the GlobalProtect app for macOS version 5.1.1 could not be properly installed because the GlobalProtect service failed to launch.
GPC-10380	Fixed an issue where the GlobalProtect app on macOS displayed the following error message when all the gateways were configured as Manual Only priority: Could not connect to Gateway, Contact your IT administrator With this fix, the app now displays the following message: Please select a gateway to connect manually
GPC-10341	Fixed an issue on Windows endpoints where, after the endpoint woke up from sleep mode, the GlobalProtect app was disconnected and then attempted to reconnect to the portal or gateway.
GPC-10311	Fixed an issue where, when the GlobalProtect app was installed on macOS and Windows, cookie authentication was successful even when the wrong password was used and GlobalProtect was still connected after users sign out of the app. With this fix, authentication cookies are now deleted from the system when users sign out of the app.
GPC-10288	Fixed an issue where, when GlobalProtect was installed using the Windows Installer (Msiexec) with on-demand as the connect method, GlobalProtect automatically tried to connect to the portal.
GPC-10261	Fixed an issue where the GlobalProtect app displayed the customized Captive Portal Detection Message in the wrong format when a different language was used other than English.
GPC-10227	Fixed a connectivity issue where, when the GlobalProtect app was installed for macOS Catalina, the GlobalProtect connection was periodically lost.
GPC-10228	Fixed an issue where the GlobalProtect app detected the presence of a captive portal even though it was not present.
GPC-10118	Fixed a periodic issue where the GlobalProtect tunnel failed to be restored after waking up from sleep mode. This issue occurred when on-demand was used as the connect method.
GPC-10024	Fixed an issue where, after upgrading to GlobalProtect 5.0.6, the GlobalProtect HIP check did not detect that Symantec Endpoint Protection 14.2 real-time protection was enabled, which caused the device to fail the HIP check.
GPC-10190	Fixed an issue where the GlobalProtect app on macOS failed to find the correct certificate for authentication to the gateway, when the object identifier (OID) was specified in the plist. With this fix, when you provide the Key Usage OID in the plist, the GlobalProtect app uses the correct certificate.
GPC-9913	Fixed an issue where the portal configuration selection criteria failed when the certificate was signed with the version 2 template.
GPC-9779	Fixed an issue that caused the GlobalProtect app to install a default route with the same metric as the system default route when split-tunneling based on access route and destination domain was enabled. This issue caused some excluded traffic to go through the tunnel.
GPC-9730	Fixed an issue where GlobalProtect failed to connect to the external gateway when the proxy was not reachable outside of the corporate network until the GlobalProtect service or the desktop was restarted.
GPC-9500	Fixed an issue in GlobalProtect for macOS endpoints where installing or upgrading the package using a Mobile Device Management (MDM) solution such as JAMF Pro resulted in a GlobalProtect app initialization failure.

Re: Troubleshooting random GlobalProtect disconnects...

devavratoka — Wed, 20 May 2020 17:59:01 GMT

This exact thing is happening to us. What's weird is we switched from GP on prem to Prisma and it started with Prisma. We disabled UDP on the RDP client on everyone's PC in the reg key settings and it seemed to reduce the number of disconnects, but they're still happening.

Re: Troubleshooting random GlobalProtect disconnects...

DanMarchewka — Wed, 20 May 2020 18:10:18 GMT

It's very frustrating. It started happening to us as soon as we put the Palo in 2+ weeks ago. We are running PanOS 9.0.6 and GP 5.1.1.

What versions are you running?

Re: Troubleshooting random GlobalProtect disconnects...

devavratoka — Wed, 20 May 2020 18:13:29 GMT

Yeah! And TAC can’t see anything unusual either.
we’re on prisma and GP 5.0.9-15

Re: Troubleshooting random GlobalProtect disconnects...

DanMarchewka — Wed, 20 May 2020 18:17:40 GMT

Yeah, my ticket with TAC has been open for a week now and it's been crickets. If I hear something I will post on here. Can you please do the same?

Re: Troubleshooting random GlobalProtect disconnects...

devavratoka — Wed, 20 May 2020 18:35:12 GMT

Absolutely. Btw, try disabling UDP on the RDP client and see if it helps you. It reduced the number of disconnects for us.

check out https://getadmx.com/?Category=Windows_10_2016&Policy=Microsoft.Policies.TerminalServer::TS_CLIENT_TURN_OFF_UDP

Re: Troubleshooting random GlobalProtect disconnects...

DanMarchewka — Wed, 20 May 2020 18:47:27 GMT

We can look into it, but it's only happening on our BYOD policy using personal PCs. Our company laptops aren't affected since they aren't RDP'ing into a work PC, just connecting to the network. The issue when using a personal device to connect via GP and then RDP to their work PC at the office. That is when we get the freezing and have to disconnect and reconnect. Since these are personal devices, it would be hard to get everyone to configure turning UDP for RDP to disabled.

Re: Troubleshooting random GlobalProtect disconnects...

jrauman — Sun, 24 May 2020 16:53:36 GMT

Hi. Original Poster here. We have made some progress on these issues. It appears there were two separate issues at play.

1. Remote Desktop Connection freezing up several times a day (up to 8-12 times for some users).

2. GlobalProtect disconnecting suddenly (which causes RDC to disconnect, not freeze)..

For #1, we dont think it had anything to do with GlobalProtect. We found that users who had Wi-Fi connections were the ones with the RD freezing up. Our resolution included ensuring the user was on 5GHz if available, and that their 5G wi-fi network didnt have any interference from neighboring wireless networks (we used Wi-Fi Analyzer from the Microsoft Store to find any). If there was another 5G WiFi network on the same channel as the user's network, we asked them to try to move their WiFi network to another channel that wasnt being used by anyone else nearby. This worked really well for many users. We did find one exception. For users who have Roku, Chromecast, Amazon FireTV type devices, those devices will create a separate WiFI network on the same channel as the WiFi network itself (i dont know why). If someone else uses those devices, RDC will freeze up. I would have users unplug those devices during working hours with good success. I reached out to Roku but they said "nothing we can do about it" and didnt explain why it was ncessary to throw up a separate network on the same channel and not at least give users the option to move the Roku network to another channel. Oh well. (NOTE: we also bought some really long ethernet cables for users with this scenario if they couldnt do anything about Wi-FI interferrence. That helped tremendously).

2. For the GP disconnects, one thing we found was that GP was set to the default timeout of 8 hours. So every day around 3-5pm, users GP would disconnect (which was before they were done workign for the day). We changed that to 12 and that issue went away. We are now seeing very few GP disconnects at other times. I'm going to do some testing this weekend on my machine to see if GP will disconnect over Ethernet over a period of a day. I'll keep you posted.

Oh, someone mentioned disabling UDP for RDC. We never had to do that to get great performance from RDC. See #1 for our resolution. Also we are trying to get Softphones to work over RDC and SIP traffic uses UDP if i recall correctly so that wouldnt work for us anyway.

Re: Troubleshooting random GlobalProtect disconnects...

devavratoka — Mon, 25 May 2020 00:34:36 GMT

@jrauman

thanks. Our problems with RDP began after users switched from GP on prem to Prisma Access. Never had disconnects on their home network before.

Re: Troubleshooting random GlobalProtect disconnects...

DanMarchewka — Tue, 26 May 2020 13:27:18 GMT

Like Devavrotoka, our problems only started happening after we put in our Palo firewall (not Prisma, but an HA Pair of 3220's) a few weeks back. Never had a single issue with RDP freezing from 10+ years on a SonicWall. It's definitely an issue with either GP or PanOS or perhaps even both.

Re: Troubleshooting random GlobalProtect disconnects...

Karl_Linder — Thu, 04 Jun 2020 08:04:02 GMT

Have the same issue, what I can see is that it´s when you start a new RDP session that make the GP stops working. But it´s not all sessions that get dropped out only a few and as time pass more and more sessions drops out.

/Karl

Re: Troubleshooting random GlobalProtect disconnects...

Chad_Parrish — Wed, 24 Jun 2020 13:45:56 GMT

We are seeing at least a few people have more frequent disconnects with GP 5.0.9-15. We haven't been able to find the issue yet. The 4.1.X branch that we were previously on kept a more stable connection.

Re: Troubleshooting random GlobalProtect disconnects...

fhewiufhwefhwe — Thu, 02 Jul 2020 16:56:02 GMT

I am trying a possible solution from another user's post after being unsuccessful resolving this for a while

https://live.paloaltonetworks.com/t5/general-topics/rdp-freeze-fix-globalprotect/m-p/335816/highlight/false#M84643

They said the following:

We struggled with the RDP freezing issue with GlobalProtect for a long time. The initial "fix" was to disable UDP for RDP in the registry. This fixed the issue for many users but also slowed down the RDP performance. We thought the issue was with GlobalProtect but after troubleshooting with Palo Alto we were able to see that at some point the remote PC just simply stopped sending RDP packets.

We opened a case with Microsoft and they gave us a registry fix to try that fixed all our RDP freeze issues and allowed us to re-enable UDP for RDP.

Important: This regedit goes on the machine you are remoting into, not the machine you are remoting from.

HKLM\SOFTWARE\Microsoft\Terminal Server Client

UseURCP (Create this new DWORD with value of 0)

You can use this from a command prompt as long as you have admin privileges on the box:

REG ADD "HKLM\SOFTWARE\Microsoft\Terminal Server Client" /v UseURCP /t REG_DWORD /d 0 /f

If you previously disabled UDP for RDP you can re-enable it.

Re: Troubleshooting random GlobalProtect disconnects...

David_Avery — Wed, 22 Jul 2020 13:42:21 GMT

Thanks Fhewiufhwefhwe,

Your reg key seems to fix the issue. While my RDP client feels more sluggish I'm not experiencing the Freeze anymore.

To All, if you have a rule which only allows [ms-rdp {app-default}], you should consider adding an [any {udp-3389}] rule below it, this initially helped make my problem less frustrating but I would still experience freezes if I did the below:

If this helps anyone from Palo Alto Networks support if they come across this issue, I found in pcaps at the firewall that it was dropping RST packets from the remote terminal server (my windows 10 desktop in this case) when under heavy screen redraw load while trying to scroll wheel through either the start menu, a folder with large images and the view set to Extra Large Icons, or the way I could always force it to freeze would be to grab a full screen window such as my browser open to my pan firewall UI and flail it around the screen wildly. I would see a massive amount of TRANSMIT and FIREWALL stage (as expected with RDP Experience options set to everything on), then when it froze about three dozen of TCP,RST packets capped in the DROP stage.

In looking at the session through a second VDI connection to our Horizon View I noticed the UDP session showed this:

c2s flow:
source: <rd> [user_vpn-L3]
dst: <rd>
proto: 17
sport: 55450 dport: 3389
state: ACTIVE type: FLOW
src user: <rd>
dst user: <rd>

s2c flow:
source: <rd> [Inside-L3]
dst: <rd>
proto: 17
sport: 3389 dport: 55450
state: ACTIVE type: FLOW
src user: <rd>
dst user: <rd>

DP : 2
index(local): : 1203093
start time : Wed Jul 22 05:38:09 2020
timeout : 36000 sec
time to live : 35992 sec
total byte count(c2s) : 330115
total byte count(s2c) : 52126669
layer7 packet count(c2s) : 1926
layer7 packet count(s2c) : 42793
vsys : vsys1
application : ms-rdp
rule : davery-wfh-limited
service timeout override(index) : False
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
layer7 processing : completed
URL filtering enabled : True
URL category : any
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : True
captive portal session : False
ingress interface : tunnel.<rd>
egress interface : ae2
session QoS rule : N/A (class 4)
tracker stage l7proc : ctd err sw

end-reason : unknown

Sometimes I would see an unknown-udp session start on the same rule, as I suspect the client side was trying a new session start but most times I did not see that though.

Normally I would expect my second rule which follows this one 'davery-wfh-limited-1' which is not app-specific on port udp-3389 to pickup the traffic that was unknown-rdp and allow but it seems like whatever is happening in the packets that the appID engine is hitting a problem identifying and then the stream breaks. Since it's udp the client would just keep sending AND since the firewall was dropping the server side TCP RST packets, the client side was unaware the server side wants the session to disconnect and start over.

I've been dealing with this problem since probably October 2019 and my users have been very frustrated to say the least. It took me about 2 months to find that I needed the non-app specific udp-3389 rule, and most clients we ended up turning off UDP. Thanks again Fhewiufhwefhwe, and thanks to the solution OP Brianhill88