GlobalProtect Slowness

[ce1028]

Hello,

I recently started a new job and have been thrown right into the fire.  Users are complaining about very slow connections from globalprotect.  They get speed tests between 3mbps - 20mbps.  Internet speed from ISP is 500Mbps.  When I attempt from a speed test site, I get a little over 100Mbps off the network but around 20Mbps when I'm on GlobalProtect.  This is not split tunnel.  Globalprotect connections are IPSec VPN

I don't want to jump to conclusions but I believe the issue is inadequate hardware.  Firewall is a PA-3050.  When I check the specs, I see max IPsec throughput is 500Mbps.  There are over 100 users connecting to globalprotect during peak times.  Assuming my understanding is correct, those 100 users are going to be sharing the 500 Mbps throughput?  Plus the profiles attached to the security policy rules (av, threat, url, decryption) add some overhead, I'm not entirely sure how much that would impact though.   The firewall also has some site-to-site VPNs too.  Would Globalprotect share the 500Mbps throughput with those Site-to-Site VPNs too, or is that 500Mbps per tunnel interface?  

I've also heard similar complaints from other Palo customers who blame the issue on globalprotect, but I'm not sure if there is truth to that, so I don't want to assume

Any advice would be helpful

[jdelio]

When trying to troubleshoot slowness, a lot of things can affect the speed.

Max Throughput .. not sure if "per tunnel" or total for machine..  hard to say.

The Policies can affect things, but also the versions can have a huge effect on everything. 

What versions are we dealing with here? PAN-OS and GP version? 

Also, just incase anyone needs it, this is a great document to help troubleshoot GlobalProtect..

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClkBCAS

[ce1028]

Thanks for reply.

In this case, version does not appear to be relevant. Before I joined the previous engineer upgraded firewall and gp version to 9.0.11 and 5.1.7.  Apparently, issue has been going on since 8.1 days from what I gather

[MP18]

@ce1028 

One thing first I will recommend to upgrade OS on firewall to 9.1.7 and GP to version 5.1.8.

Are you doing any ssl decryption on the firewall?

Check the firewall interface for any errors or speed duplex mismatch?

Does firewall has direct connection to the internet?

Regards

[svintinner]

For the guys that have replied, I'm curious what kind of performance you see on your GlobalProtect sessions?  I think it might be helpful to set a baseline when talking about GlobalProtect performance.  I've read tons of these posts on the forums, but rarely see anyone discuss what we should expect.

In my testing I can never average more than 50-70 mbps GlobalProtect SSL VPN connection (dedicated 3020 firewall with just me,  dedicated 1 Gbps internet link on both sides for just me, 30ms latency, no inspection or app-id, no QoS, iperf3).  I can open a second SSL VPN connection from a different computer and simultaneously get another 50-70 mbps without impacting the first session.  I don't see a significant CPU load on the firewall at either point.  I can do testing outside GlobalProtect (static NAT) and pretty consistently get 940 mbps.  My assumption is that this is some internal tuning limitations that we can't see.

On my production system, I will have stretches where I can get 50-70 mbps, but this will frequently drop down to the 2-10 mbps range (for minutes at a time).  Like the OP, the overall bandwidth usage doesn't explain all of the issues).   Certainly, I can see slowness when there are peaks in bandwidth usage, but I also see slowness that doesn't correspond to any bandwidth usage.  My assumption is that it is due to firewall load (although the firewall doesn't show 100% CPU, I assume the GP process is somehow throttled and that the performance slowness is due to other stream processing inspections and app-id that is happening).

I can run a simultaneous test (iperf3) where I test using a static NAT (non-GP) at 200 mbps, along side 2 GP connections.  The static NAT connection will remain consistent, while the two GP connections will suffer performance hits around the same time.

I should note that I've read the usual comments about SSL VPN and performance (due to a TCP session encapsulated in another TCP session).   I can see this demonstrated when I do testing at my DR site and I run into (what I assume) are throttling issues when the interior and exterior TCP sessions have conflicting sliding windows.  For example, the session will be cooking along at 70mbps for 30 seconds, then drop to zero and then ramp back up to 70 mbps.   I'm planning to do some testing on my test site with GlobalProtect in IPSEC mode to see if this goes away or if my overall bandwidth is improved.

Anyway here are some things I've noticed (recognizing this is the blind leading the blind), in case any of it give you some things to check on your system:

• As mentioned above, no matter what your bandwidth, Globalprotect seems to have other limitations, so setting expectations with users is critical.  (i.e. a Speedtest is never going to show the full bandwidth).
• Regular SMB performance is just awful on SSL VPN.  It works better if you use SMBv3, so ensure that your clients and file servers are upgraded.
• In my testing just enabling QoS on an interface caused significant performance hit on GlobalProtect.
• Security policies (inspections) can impact performance.  Try tuning them or turning them off (while testing).
• Verify you aren't having fragmentation issues.  GP 5.2.5 supports changing the MTU size.

If anyone has any better information, especially about the internal workings or scheduling of GP traffic inside the firewall, I'd love to hear it.

[sebastianvd]

We had lots of speed issues which boiled down to :

- use IPSEC instead of SSL

- check your MTU/fragmented packets (Azure drops out of order packets, limits MTU to 1400 and our local WIFI L3 roaming feature took some MTU size extra away).

[svintinner]

Just a follow up on my post.  We enabled IPSEC on GlobalProtect and all our slowness issues were resolved (we get nearly full bandwidth).   I still find it hard to believe that SSL VPN performance is so terrible and that Palo Alto is happy with it, but we've moved on to just requiring IPSEC.

[m.fusoni]

I believe Palo Alto is happier with IPsec, indeed it's the dafault transport method that GP tries when connecting to the gateway.

[MP18]

@svintinner  Also IPSEC is UDP it has less overhead as compare to SSL.

Thanks for updating the Community

Regards

[FBMTRAV]

I recently found about 30 of my 4,000+ users have crawling speeds (2Mbps) over Global Protect using IPsec when compared to AnyConnect over SSL (30Mbps). During troubleshooting we are certain their ISP is throttling or applying DPI to their IPsec tunnel. To work around their ISP issue, we added the ability for them to connect to Global Protect over SSL (a checkbox in the client settings). Now they are seeing the same speeds as AnyConnect. This may or may not be related to your problem, but something to try in case SSL works faster for them compared to IPsec.