Troubleshoot Performance Related to GlobalProtect

Palo Alto Networks understands that with an increased remote workforce, there is the possibility of performance issues in your network with GlobalProtect. Here is some great information on how to troubleshoot performance related to GlobalProtect.

First of all, please bear in mind that SSL VPN is not designed to be efficient (it is best effort and not designed for high throughput) mostly because it's TCP-based (TCP inside of TCP). What that means is that the GlobalProtect (GP) client and/or firewall doesn't have control over the underlying TCP layer (error detection, flow control, congestion control), which is slowing down the throughput. That is the main reason why we would recommend, if you're experiencing slow throughput (when using SSL), to migrate to IPsec whenever possible.

Below are a few ideas/suggestions/limitations regarding this topic.

GlobalProtect client-related issues (i.e., slow throughput when using GlobalProtect client)

It is expected for the throughput to be slower when the GlobalProtect client is being used as opposed to non-VPN or direct connection. There are several reasons for that:

Both the client and the firewall have to encrypt/decrypt and encapsulate/decapsulate traffic, which will introduce processing delays
Time needed for the operation depends on the current client/firewall load
TCP inside TCP encapsulation (no control over underlying TCP layer)

It will be difficult to give numbers on the expected throughput difference between GlobalProtect client versus non-GlobalProtect client scenarios. The primary reason is that the throughput depends on several factors and each setup is different.

General issues, limitations and recommendations

Performance related issues are not easy to troubleshoot. In other words, the throughput depends on several factors.

With or without VPN client, you will have different download speeds measured if you try to download the same file multiple times. The network path (between client and server) usually passes several devices and links, and the conditions are changing dynamically (CPU/link utilization, load, buffers, and so on).

Always try to collect a minimum of two sets of data for "low throughput" and "high throughput" scenario, so you have a baseline that you can use to compare. Always clarify which protocols are used (smb, http, ftp, etc.), location of the clients/servers, and Internet link speeds. Again, low/high throughput scenarios could also mean using two different protocols, such as smb and ftp (please bear in mind that smb is known to have the worst performance).

Of course, you'll need to use the same infrastructure for both tests. Otherwise, you don't have valid data for comparison. It would be great if we can have tests for two different protocols (like http and smb transfer) in two different scenarios. For example, http download with or without GlobalProtect, and the same for SMB.

Please note that firewall debugs and captures are affecting the firewall's load and can have negative impact on the actual throughput and test results. Although it sounds contradictory, debugs can sometimes even improve throughput, but we are talking about corner case scenario.

Don't rely solely on speedtest.net and don't forget to check trivial issues such as interface speed/duplex mismatch.

Packet captures

1. In case of GlobalProtect client, please collect packet captures on the client side (both physical and virtual adapter) and packet captures on the server side.

2. If possible, it would be ideal to have external packet captures before and after the firewall. If it's not possible, then collect the files on the firewall. Although the firewall can do packet captures, these are not 100% reliable (considered debugging feature):

We may have some packets missing even though they have been successfully forwarded (if the firewall was loaded and missed writing a packet to a .pcap file, etc.).
Some packets may be seen as "out-of-order" in firewall captures, even though they were received in the right order.

Troubleshooting Latency

If you are troubleshooting latency or "packets out-of-order" type of issues, external packet capture is mandatory (external packet capture means before and after the firewall without intermediate devices, otherwise, there's no way to know if the problem is on the firewall side or somewhere within the routing or switching infrastructure).

As always, when it comes to packet processing, don't forget to collect:

Session IDs
"flow basic" (sometimes even 'tcp all' or 'tcp reass', but it really depends on specific situation)
Global counters
Traffic logs
Tech support file