topic Re: GlobalProtect Slowness in GlobalProtect Discussions

GlobalProtect Slowness

ce1028 — Thu, 18 Mar 2021 01:53:49 GMT

Hello,

I recently started a new job and have been thrown right into the fire. Users are complaining about very slow connections from globalprotect. They get speed tests between 3mbps - 20mbps. Internet speed from ISP is 500Mbps. When I attempt from a speed test site, I get a little over 100Mbps off the network but around 20Mbps when I'm on GlobalProtect. This is not split tunnel. Globalprotect connections are IPSec VPN

I don't want to jump to conclusions but I believe the issue is inadequate hardware. Firewall is a PA-3050. When I check the specs, I see max IPsec throughput is 500Mbps. There are over 100 users connecting to globalprotect during peak times. Assuming my understanding is correct, those 100 users are going to be sharing the 500 Mbps throughput? Plus the profiles attached to the security policy rules (av, threat, url, decryption) add some overhead, I'm not entirely sure how much that would impact though. The firewall also has some site-to-site VPNs too. Would Globalprotect share the 500Mbps throughput with those Site-to-Site VPNs too, or is that 500Mbps per tunnel interface?

I've also heard similar complaints from other Palo customers who blame the issue on globalprotect, but I'm not sure if there is truth to that, so I don't want to assume

Any advice would be helpful

Re: GlobalProtect Slowness

jdelio — Thu, 18 Mar 2021 19:39:33 GMT

When trying to troubleshoot slowness, a lot of things can affect the speed.

Max Throughput .. not sure if "per tunnel" or total for machine.. hard to say.

The Policies can affect things, but also the versions can have a huge effect on everything.

What versions are we dealing with here? PAN-OS and GP version?

Also, just incase anyone needs it, this is a great document to help troubleshoot GlobalProtect..

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClkBCAS

Re: GlobalProtect Slowness

ce1028 — Fri, 19 Mar 2021 14:14:15 GMT

Thanks for reply.

In this case, version does not appear to be relevant. Before I joined the previous engineer upgraded firewall and gp version to 9.0.11 and 5.1.7. Apparently, issue has been going on since 8.1 days from what I gather

Re: GlobalProtect Slowness

MP18 — Sat, 20 Mar 2021 03:16:37 GMT

@ce1028

One thing first I will recommend to upgrade OS on firewall to 9.1.7 and GP to version 5.1.8.

Are you doing any ssl decryption on the firewall?

Check the firewall interface for any errors or speed duplex mismatch?

Does firewall has direct connection to the internet?

Regards

Re: GlobalProtect Slowness

svintinner — Mon, 22 Mar 2021 16:31:22 GMT

For the guys that have replied, I'm curious what kind of performance you see on your GlobalProtect sessions? I think it might be helpful to set a baseline when talking about GlobalProtect performance. I've read tons of these posts on the forums, but rarely see anyone discuss what we should expect.

In my testing I can never average more than 50-70 mbps GlobalProtect SSL VPN connection (dedicated 3020 firewall with just me, dedicated 1 Gbps internet link on both sides for just me, 30ms latency, no inspection or app-id, no QoS, iperf3). I can open a second SSL VPN connection from a different computer and simultaneously get another 50-70 mbps without impacting the first session. I don't see a significant CPU load on the firewall at either point. I can do testing outside GlobalProtect (static NAT) and pretty consistently get 940 mbps. My assumption is that this is some internal tuning limitations that we can't see.

On my production system, I will have stretches where I can get 50-70 mbps, but this will frequently drop down to the 2-10 mbps range (for minutes at a time). Like the OP, the overall bandwidth usage doesn't explain all of the issues). Certainly, I can see slowness when there are peaks in bandwidth usage, but I also see slowness that doesn't correspond to any bandwidth usage. My assumption is that it is due to firewall load (although the firewall doesn't show 100% CPU, I assume the GP process is somehow throttled and that the performance slowness is due to other stream processing inspections and app-id that is happening).

I can run a simultaneous test (iperf3) where I test using a static NAT (non-GP) at 200 mbps, along side 2 GP connections. The static NAT connection will remain consistent, while the two GP connections will suffer performance hits around the same time.

I should note that I've read the usual comments about SSL VPN and performance (due to a TCP session encapsulated in another TCP session). I can see this demonstrated when I do testing at my DR site and I run into (what I assume) are throttling issues when the interior and exterior TCP sessions have conflicting sliding windows. For example, the session will be cooking along at 70mbps for 30 seconds, then drop to zero and then ramp back up to 70 mbps. I'm planning to do some testing on my test site with GlobalProtect in IPSEC mode to see if this goes away or if my overall bandwidth is improved.

Anyway here are some things I've noticed (recognizing this is the blind leading the blind), in case any of it give you some things to check on your system:

As mentioned above, no matter what your bandwidth, Globalprotect seems to have other limitations, so setting expectations with users is critical. (i.e. a Speedtest is never going to show the full bandwidth).
Regular SMB performance is just awful on SSL VPN. It works better if you use SMBv3, so ensure that your clients and file servers are upgraded.
In my testing just enabling QoS on an interface caused significant performance hit on GlobalProtect.
Security policies (inspections) can impact performance. Try tuning them or turning them off (while testing).
Verify you aren't having fragmentation issues. GP 5.2.5 supports changing the MTU size.

If anyone has any better information, especially about the internal workings or scheduling of GP traffic inside the firewall, I'd love to hear it.

Re: GlobalProtect Slowness

sebastianvd — Fri, 26 Mar 2021 08:46:56 GMT

We had lots of speed issues which boiled down to :

- use IPSEC instead of SSL

- check your MTU/fragmented packets (Azure drops out of order packets, limits MTU to 1400 and our local WIFI L3 roaming feature took some MTU size extra away).

Re: GlobalProtect Slowness

svintinner — Thu, 14 Apr 2022 18:34:34 GMT

Just a follow up on my post. We enabled IPSEC on GlobalProtect and all our slowness issues were resolved (we get nearly full bandwidth). I still find it hard to believe that SSL VPN performance is so terrible and that Palo Alto is happy with it, but we've moved on to just requiring IPSEC.

Re: GlobalProtect Slowness

m.fusoni — Wed, 01 Jun 2022 09:40:35 GMT

I believe Palo Alto is happier with IPsec, indeed it's the dafault transport method that GP tries when connecting to the gateway.

Re: GlobalProtect Slowness

MP18 — Wed, 01 Jun 2022 20:07:31 GMT

@svintinner Also IPSEC is UDP it has less overhead as compare to SSL.

Thanks for updating the Community

Regards

Re: GlobalProtect Slowness

FBMTRAV — Wed, 26 Apr 2023 16:56:02 GMT

I recently found about 30 of my 4,000+ users have crawling speeds (2Mbps) over Global Protect using IPsec when compared to AnyConnect over SSL (30Mbps). During troubleshooting we are certain their ISP is throttling or applying DPI to their IPsec tunnel. To work around their ISP issue, we added the ability for them to connect to Global Protect over SSL (a checkbox in the client settings). Now they are seeing the same speeds as AnyConnect. This may or may not be related to your problem, but something to try in case SSL works faster for them compared to IPsec.

Re: GlobalProtect Slowness

Waly — Fri, 17 Jan 2025 14:00:36 GMT

i am having a client having the same issue, 10% of the users are facing slowness while 90% are not facing it using IPSec tunnel, while disabling the IPSec and using only SSL it is working fine.

Re: GlobalProtect Slowness

MP18 — Tue, 21 Jan 2025 17:53:02 GMT

@Waly Normally SSL works slower as it is TCP.IPSEC gives faster speed.

We have both enabled and see connection is slower when using SSL and this is expected behavior.

Regards