topic Re: Slow speed with GlobalProtect in General Topics

Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 17:26:57 GMT

Hi to all,

We are trying to understand why the download speed is really slow vía GP.

We stablish a VPN GP with IPsec without Split Tunneling. We acces to some public web to download a test file. Im downloading a 1G file.

If we download without GP but through the Palo Alto we achieve 60MB/s, but vía GP we achieve maybe 6 or 10MB/s.

Any idea about that? Someona has experienced this issue? The differents shows issued don't present mss or mtu problems.

Thank you!

Re: Slow speed with GlobalProtect

DigitalAffinity — Wed, 10 Apr 2019 18:08:35 GMT

Which model of firewall are you using?

Do you have threat profiles enabled for that traffic?

Each firewall model has a throughput limit based on the horsepower of the firewall. If you run traffic through the threat stack, it slows it down more. For example, I had a PA-500 and the max throughput is rated at 100mbps for traffic running through the threat stack. That equates to about 10/MB a second. Also, you have overhead from the IPSec encryption, that will slow down most traffic as well.

It is really the price you pay for security. Running traffic through the VPN and firewall means that additional overhead gets added on the traffic. Here are some ideas if you want:

- enable some routes in GP to exclude traffic to certain sites from the VPN

- If you have a GP portal license, you can turn on "exclude video" sites and it won't slow down video.

- create policies to disable the threat stack for certain type of traffic

Re: Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 18:59:43 GMT

The PA850 sheet talks about 500Mbps (62MB/s) IPSec throughput. There are NO profiles attached to the rule, the throughput following the datasheet without profiles are about 2Gbps...

If you hace 100mbps as you said this is 12MB/s so if you are receiving 10MBs this is acceptable.

So, i don't understand where the bottleneck happens. It's nothing related about price fot security.. it's just specs.

Thank you,

Re: Slow speed with GlobalProtect

DigitalAffinity — Wed, 10 Apr 2019 19:13:34 GMT

I understand. The specs for your firewall are high enough. I was just relating my experience where traffic outside of the GP VPN was fast, but traffic over the VPN is slow. That seems to be a universal condition introduced by the overhead of the VPN encryption. That is what I meant by price of security; VPN traffic won't be as fast as regular firewall traffic in my experience. Maybe there is some setting that makes it faster, but I'd love to know that as well. So far across two different PA firewalls, VPN traffic is slower.

Re: Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 19:17:35 GMT

Let's if someone can add more light to this subject. I know about the overhead, for this reason I check-in MTU or MSS but the performance it's not acceptable. Datasheet says 500mbps... The are a lot of difference between 62MBs expected vs 5-6MBs...

Thank you,

Re: Slow speed with GlobalProtect

Brandon_Wertz — Wed, 10 Apr 2019 19:42:35 GMT

So as my recent efforts have lead me down this path, this is a very difficult question to answer.

When I first deployed GP using SSL only type tunnel on my 5220 (With decryption) I noticed a significant disparity in my speedtest results. At home without GP I would get between 90-105Mbps. With GP SSL type VPN tunnel I would get between 10-25Mbps. I opened a few tickets with TAC and got various answers and "fixes."

The best fix I've since deployed is converting the tunnel type to allow IPSec VPN for clients. This setting got me to around 50-65Mbps on a speedtest.

In one of my tickets with TAC they modified the tunnel I use for VPN to a lower amount to account for the encryption overhead (something I hadn't done.) There were varied settings here going down to 1424 down to as low as 1350 which is where I'm currently at. This was done to help prevent fragmentation of the "transfer stream." When this particular change was made I didn't really notice a significant change in throughput capacity.

In one of my tickets an engineer made a great observation, speed test sites aren't really a good measure of "throughput" or capacity while on VPN. They have their own unique setup that doesn't necessarily jive well for a VPN client.

Your best bet is to use a site that tests as if you're transferring a file. So in my case I feel getting 50-65Mbps via VPN w/o split-tunneling is more than enough.

Here is a better site to use: https://www.thinkbroadband.com/download

Re: Slow speed with GlobalProtect

Brandon_Wertz — Wed, 10 Apr 2019 19:32:58 GMT

If you're trying to do this on a PA-850 and expecting high throughput I wouldn't. There are A LOT of factors that go into a devices performance specs.

I'm assuming your box is doing more than just supporting this single user GP connection. If your client is getting 48Mbps (6MB) of total potential capacity of 500Mbps, what other traffic is the box doing?

Re: Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 20:04:22 GMT

Hi,

Thank you for you reply.

This device is in test environment prior to full migration. So unfortunatelly yes, no other traffic is crossing the firewall.

I don't trust on those speedtest, the test im doing it's director download vía browser from some website that offers Free test downloads.
Where you tune the tunnel capacity?

Regards,

Re: Slow speed with GlobalProtect

gwesson — Wed, 10 Apr 2019 21:59:15 GMT

The testing you are doing is flawed, at least when compared with a full traffic test. The reason is based on the number of sessions and how they are handled within the dataplane of the firewall.

If you're using a single download from your browser, you are only using a single TCP connection to actually get the content. You'll have multiple connections when browsing the site, but that one object being downloaded is limited by many factors.

If you had 50 PCs downloading 50 different objects from different servers, then you'll have a much more accurate test. Sessions can be distributed across dataplanes and cores, for example.

Re: Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 22:06:57 GMT

Hi,
Thank you for your response. Im try to explain better the scenario.

1st Test. One computer behind the firewall (Site A) downloading a file from https://www.thinkbroadband.com/download I achieve acceptable speed. Im the only one using the firewall, no security profiles attached to the rule.

2n Test. From a computer in a remote site (Site B) and download the same file, I achieve better results due to the best line in Site B.

3rd Test From Site B establish a GP IPsec ti Site A without Split tunneling, so all traffic crosses the FW. I go to the same Site and download the same file, the download speed dramatically slowdown.

In 2nd and 3rd scenario I'm the only crossing the FW.

Regards,

Re: Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 22:09:14 GMT

Hi,
Thank you for your response. Im try to explain better the scenario.

1st Test. One computer behind the firewall (Site A) downloading a file from https://www.thinkbroadband.com/download I achieve acceptable speed. Im the only one using the firewall, no security profiles attached to the rule.

2n Test. From a computer in a remote site (Site B) and download the same file, I achieve better results due to the best line in Site B.

3rd Test From Site B establish a GP IPsec to Site A without Split tunneling, so all traffic crosses the FW. I go to the same Site and download the same file, the download speed dramatically slowdown.

In 2nd and 3rd scenario I'm the only one crossing the FW.

Regards,

Re: Slow speed with GlobalProtect

gwesson — Wed, 10 Apr 2019 22:27:27 GMT

In all three scenarios, you're still using a single download. There are bottlenecks that can be associated with a single session (and even more when it comes to IPSec). Things like encapsulation/decapsulation and how it is split across multiple cores, and a whole host of things.

The datasheet numbers are for scale, not for individual downloads from a single client.

I don't want to come across as harsh, I just can't think of a better way to say it. A single download is VASTLY different than real clients connecting and doing regular work.

If you insist on using this type of flawed testing, at the very least try doing several downloads at the same time from different sites on that test client at site B. At least then you'll be using multiple sessions to fill the tunnel better.

Re: Slow speed with GlobalProtect

nanukanu — Wed, 10 Apr 2019 22:52:37 GMT

Hi Gwesson,

Thank you for your words,

I understand how It works, thank you for your explanation.
But the fact is that the test that I am doing is the way of working that our customer has.
Workers in remote sites download files via HTTP and via FTP through Global Protect and L2L VPN and it is difficult to download the same file in múltiple parts as you are suggesting.

I am just surprised by the degradation of speed in a device with these characteristics, when the current equipment, much more older (ASA FW) and through VPN supports speeds by far much better doing the same.

Appreciate for your help,

Thank you,

Re: Slow speed with GlobalProtect

Adam42 — Mon, 27 Jan 2020 14:17:28 GMT

Hello @nanukanu I hope you did find a solution for your issue

I'm facing the same issue using PA-200 but the Upload is quite good but the download traffic is slowing down (from 30 Mbps to 2Mbps).

SpeedTest Without Firewall 30 Mbps

SpeedTest with Firewall PA-200 2Mbps

Re: Slow speed with GlobalProtect

HenriqueGurgel — Wed, 04 Mar 2020 15:04:14 GMT

Hi @Adam42 ,

I am facing a very similar issue as you are.

Please could you tell which version of the firewall and the global protect you are using. I am suspecting of some version bug and I would like to know if you the version you are using is the same I am.

Thanks

Re: Slow speed with GlobalProtect

DigitalAffinity — Wed, 04 Mar 2020 19:35:05 GMT

@HenriqueGurgel I think the slowdown issue is inherent in Palo Alto Networks GlobalProtect. I have used PAN GP for about 6 years now, across two different models (PA-500 and now PA-820) and the situation with the degraded download speed has always been the same. The firewall software and GP client version do not seem to matter. I've seen this same issue on PAN 6, 7, 8, and now 9 versions, and GP client 2, 3, 4, and now 5. We have a 1GB fiber line to the PA-820. We have people working remotely from customer offices, on all types of different connections (wired, wireless, 200mbps, 500mbps, 1gbps, it doesn't matter). Download tests will always report slow speeds (2-4mbps).

However, as was mentioned by another reply, when you look at the performance in aggregate, meaning across all remote users, the performance isn't as slow as 2-4mbps. I think part of the issue is the speedtests; I don't think they play well with IPsec VPN in general. However, I agree that other vendors' firewall VPNs (Cisco, SonicWall, etc.) do not seem to suffer from the same slowdowns. But remember also that you might have the threat stack enabled so the traffic has additional processing done on it. I realize Palo Alto Networks says that shouldn't slow down traffic below a certain amount, but here we are, all posting about slow PAN GP speeds.

Over the years, I've never bothered to open a support case to get the official answer, as the speed was always "good enough" to support our workloads. However, I would like to understand why this happens.

Re: Slow speed with GlobalProtect

S.Swindall — Thu, 16 May 2024 21:25:04 GMT

The best way to test true throughput is to download and run "openspeedtest" on a server on the same switch or local LAN with your firewall. It will setup a speed test page on any server or workstation and the user just hits the start button. That will give you actual speed between the client and local server cutting out all other delays. The link below will take you to the windows version of the download page they also have versions for different operating systems. We use this to test speed instead of IPERF3 now. Works great and any user can figure it out.

https://go.openspeedtest.com/Win