Troubleshooting random GlobalProtect disconnects...

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Reply
Highlighted
L1 Bithead

Yeah, my ticket with TAC has been open for a week now and it's been crickets.  If I hear something I will post on here.  Can you please do the same?

Highlighted
L1 Bithead

Absolutely. Btw, try disabling UDP on the RDP client and see if it helps you. It reduced the number of disconnects for us. 

check out https://getadmx.com/?Category=Windows_10_2016&Policy=Microsoft.Policies.TerminalServer::TS_CLIENT_TU...

Highlighted
L1 Bithead

We can look into it, but it's only happening on our BYOD policy using personal PCs.  Our company laptops aren't affected since they aren't RDP'ing into a work PC, just connecting to the network.  The issue when using a personal device to connect via GP and then RDP to their work PC at the office.  That is when we get the freezing and have to disconnect and reconnect.  Since these are personal devices, it would be hard to get everyone to configure turning UDP for RDP to disabled.

Highlighted
L1 Bithead

Hi. Original Poster here. We have made some progress on these issues. It appears there were two separate issues at play.

1. Remote Desktop Connection freezing up several times a day (up to 8-12 times for some users).

2. GlobalProtect disconnecting suddenly (which causes RDC to disconnect, not freeze)..

 

For #1, we dont think it had anything to do with GlobalProtect.  We found that users who had Wi-Fi connections were the ones with the RD freezing up. Our resolution included ensuring the user was on 5GHz if available, and that their 5G wi-fi network didnt have any interference from neighboring wireless networks (we used Wi-Fi Analyzer from the Microsoft Store to find any).  If there was another 5G WiFi network on the same channel as the user's network, we asked them to try to move their WiFi network to another channel that wasnt being used by anyone else nearby.  This worked really well for many users.  We did find one exception. For users who have Roku, Chromecast, Amazon FireTV type devices, those devices will create a separate WiFI network on the same channel as the WiFi network itself (i dont know why). If someone else uses those devices, RDC will freeze up.  I would have users unplug those devices during working hours with good success. I reached out to Roku but they said "nothing we can do about it" and didnt explain why it was ncessary to throw up a separate network on the same channel and not at least give users the option to move the Roku network to another channel. Oh well.  (NOTE: we also bought some really long ethernet cables for users with this scenario if they couldnt do anything about Wi-FI interferrence.  That helped tremendously).

 

2. For the GP disconnects, one thing we found was that GP was set to the default timeout of 8 hours.  So every day around 3-5pm, users GP would disconnect (which was before they were done workign for the day).  We changed that to 12 and that issue went away.  We are now seeing very few GP disconnects at other times.  I'm going to do some testing this weekend on my machine to see if GP will disconnect over Ethernet over a period of a day.  I'll keep you posted.

 

Oh, someone mentioned disabling UDP for RDC.  We never had to do that to get great performance from RDC.  See #1 for our resolution.  Also we are trying to get Softphones to work over RDC and SIP traffic uses UDP if i recall correctly so that wouldnt work for us anyway.

Highlighted
L1 Bithead

@jrauman  

thanks. Our problems with RDP began after users switched from GP on prem to Prisma Access. Never had disconnects on their home network before. 

Highlighted
L1 Bithead

Like Devavrotoka, our problems only started happening after we put in our Palo firewall (not Prisma, but an HA Pair of 3220's) a few weeks back.  Never had a single issue with RDP freezing from 10+ years on a SonicWall.  It's definitely an issue with either GP or PanOS or perhaps even both. 

Highlighted
L0 Member

Hi

 

Have the same issue, what I can see is that it´s when you start a new RDP session that make the GP stops working. But it´s not all sessions that get dropped out only a few and as time pass more and more sessions drops out.

 

/Karl

Tags (1)
Highlighted
L1 Bithead

We are seeing at least a few people have more frequent disconnects with GP 5.0.9-15.  We haven't been able to find the issue yet.  The 4.1.X branch that we were previously on kept a more stable connection.

Highlighted
L3 Networker

I am trying a possible solution from another user's post after being unsuccessful resolving this for a while

https://live.paloaltonetworks.com/t5/general-topics/rdp-freeze-fix-globalprotect/m-p/335816/highligh...

 

They said the following:

 

We struggled with the RDP freezing issue with GlobalProtect for a long time.  The initial "fix" was to disable UDP for RDP in the registry.  This fixed the issue for many users but also slowed down the RDP performance.  We thought the issue was with GlobalProtect but after troubleshooting with Palo Alto we were able to see that at some point the remote PC just simply stopped sending RDP packets.

We opened a case with Microsoft and they gave us a registry fix to try that fixed all our RDP freeze issues and allowed us to re-enable UDP for RDP.

 

Important: This regedit goes on the machine you are remoting into, not the machine you are remoting from.

 

HKLM\SOFTWARE\Microsoft\Terminal Server Client

UseURCP (Create this new DWORD with value of 0)

 

You can use this from a command prompt as long as you have admin privileges on the box:

REG ADD "HKLM\SOFTWARE\Microsoft\Terminal Server Client" /v UseURCP /t REG_DWORD /d 0 /f

 

If you previously disabled UDP for RDP you can re-enable it.

Highlighted
L2 Linker

Thanks Fhewiufhwefhwe,

 

Your reg key seems to fix the issue. While my RDP client feels more sluggish I'm not experiencing the Freeze anymore.

 

To All, if you have a rule which only allows [ms-rdp {app-default}], you should consider adding an [any {udp-3389}] rule below it, this initially helped make my problem less frustrating but I would still experience freezes if I did the below:

 

If this helps anyone from Palo Alto Networks support if they come across this issue, I found in pcaps at the firewall that it was dropping RST packets from the remote terminal server (my windows 10 desktop in this case) when under heavy screen redraw load while trying to scroll wheel through either the start menu, a folder with large images and the view set to Extra Large Icons, or the way I could always force it to freeze would be to grab a full screen window such as my browser open to my pan firewall UI and flail it around the screen wildly. I would see a massive amount of TRANSMIT and FIREWALL stage (as expected with RDP Experience options set to everything on), then when it froze about three dozen of TCP,RST packets capped in the DROP stage.

 

In looking at the session through a second VDI connection to our Horizon View I noticed the UDP session showed this:

c2s flow:
source: <rd> [user_vpn-L3]
dst: <rd>
proto: 17
sport: 55450 dport: 3389
state: ACTIVE type: FLOW
src user: <rd>
dst user: <rd>

s2c flow:
source: <rd> [Inside-L3]
dst: <rd>
proto: 17
sport: 3389 dport: 55450
state: ACTIVE type: FLOW
src user: <rd>
dst user: <rd>

DP : 2
index(local): : 1203093
start time : Wed Jul 22 05:38:09 2020
timeout : 36000 sec
time to live : 35992 sec
total byte count(c2s) : 330115
total byte count(s2c) : 52126669
layer7 packet count(c2s) : 1926
layer7 packet count(s2c) : 42793
vsys : vsys1
application : ms-rdp
rule : davery-wfh-limited
service timeout override(index) : False
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
layer7 processing : completed
URL filtering enabled : True
URL category : any
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : True
captive portal session : False
ingress interface : tunnel.<rd>
egress interface : ae2
session QoS rule : N/A (class 4)
tracker stage l7proc : ctd err sw

end-reason : unknown

 

Sometimes I would see an unknown-udp session start on the same rule, as I suspect the client side was trying a new session start but most times I did not see that though.

 

Normally I would expect my second rule which follows this one 'davery-wfh-limited-1' which is not app-specific on port udp-3389 to pickup the traffic that was unknown-rdp and allow but it seems like whatever is happening in the packets that the appID engine is hitting a problem identifying and then the stream breaks. Since it's udp the client would just keep sending AND since the firewall was dropping the server side TCP RST packets, the client side was unaware the server side wants the session to disconnect and start over.

I've been dealing with this problem since probably October 2019 and my users have been very frustrated to say the least. It took me about 2 months to find that I needed the non-app specific udp-3389 rule, and most clients we ended up turning off UDP. Thanks again Fhewiufhwefhwe, and thanks to the solution OP Brianhill88


--Why so many drops! Firewall stop telling me I made the rule wrong and tell me how to fix it :-P
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!