PA-300 induced high latency for GP clients with just 200Mbps troubput

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

PA-300 induced high latency for GP clients with just 200Mbps troubput

L3 Networker

Our users on Global Protect clients downloading a somewhat large file all at the same time. The aggregate amount download was 200GB over a couple of hours. The server they were downloading from was outside of our data center and the files were going through Global Protect because the destination server was part of the split tunnel. During this period latency for the from client to data center increased from a normal 30ms to 1s. This caused SQL performance degradation. The VM-300 is rated for like 1.7Gbps throughput. I don't know if this 200Mbps throughput would have the impact if users had been downloading from our data center as opposed to an outside server. Any thoughts on how this kind of traffic might have impacted latencies for clients to the data center via Global Protect? 
 

1 accepted solution

Accepted Solutions

Cyber Elite
Cyber Elite

Palo has performance tuning suggestions "Performance Tuning of the VM-Series for ESXi".

Discuss with VMware team which of those settings are reasonable to implement.

 

https://docs.paloaltonetworks.com/vm-series/11-0/vm-series-deployment/set-up-a-vm-series-firewall-on...

Enterprise Architect, Security @ Cloud Carib Ltd
Palo Alto Networks certified from 2011

View solution in original post

6 REPLIES 6

Cyber Elite
Cyber Elite

@MichaelMedwid,

Whenever you see this issue on a VM-series one of the first things that I'm suspect of is the host. Does the VM-300 and the SQL servers reside on the same host? Do you have multiple hosts all running on a shared enclosure? What did the enclosure/host uplink interfaces report when you were seeing the issue?

L3 Networker

The SQL server and VM PAN are on separate hosts. CPU of the host and guest were both calm at 45%. There are other guests on the (Cisco UCS) host. But as I say there's no spiking during the problem period observed. VM latency and IOPs normal. The uplinks are at least 2x the consumed capacity at their narrowest point. The latency reported by the GP user was just ping latency into the data center. 

Cyber Elite
Cyber Elite

45% as average might seem ok but in virtual environment you also need to take into account that hypervisor has way easier to find available slots for VMs with lower vCPU count than ones that have more vCPUs.

How many vCPUs your VM-300 has?

How many other VMs on same host have?

Does VMware CPU Ready graph of VM-300 go up when users download files? If it does it means you don't have enough physical CPU resources to satisfy need and you need to reserve CPU for VM-300 to perform as expected.

CPU Ready shows time when VM is waiting for CPU resources but hypervisor don't have any to give.

Enterprise Architect, Security @ Cloud Carib Ltd
Palo Alto Networks certified from 2011

The VM has two CPUs. CPU 2 peaks at around 67% and the CPU 1 was 30%. This is out of Solar Winds. "CPU Peak Trend" says the VM is at 70%. There are 30 VMs  on the host. I'm not familiar with the CPU ready metric. I'll have to ask the VMW admins perhaps. 

Would it be advisable to add CPU to the VM PAN and does it require downtime?

Cyber Elite
Cyber Elite

Never overprovision VMs.

As I mentioned earlier it is way harder for hypervisor to find timeslots to provide VM physical CPU resources if VM has big amount of vCPUs or has more vCPUs than other VMs have on same host.

So by going from 2vCPU to 4 you might make things worse.

 

Investigate CPU Ready and analyze with VMware team if it is in accepted treshold.

Enterprise Architect, Security @ Cloud Carib Ltd
Palo Alto Networks certified from 2011

Cyber Elite
Cyber Elite

Palo has performance tuning suggestions "Performance Tuning of the VM-Series for ESXi".

Discuss with VMware team which of those settings are reasonable to implement.

 

https://docs.paloaltonetworks.com/vm-series/11-0/vm-series-deployment/set-up-a-vm-series-firewall-on...

Enterprise Architect, Security @ Cloud Carib Ltd
Palo Alto Networks certified from 2011
  • 1 accepted solution
  • 1801 Views
  • 6 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!