03-24-2023 10:23 AM
Our users on Global Protect clients downloading a somewhat large file all at the same time. The aggregate amount download was 200GB over a couple of hours. The server they were downloading from was outside of our data center and the files were going through Global Protect because the destination server was part of the split tunnel. During this period latency for the from client to data center increased from a normal 30ms to 1s. This caused SQL performance degradation. The VM-300 is rated for like 1.7Gbps throughput. I don't know if this 200Mbps throughput would have the impact if users had been downloading from our data center as opposed to an outside server. Any thoughts on how this kind of traffic might have impacted latencies for clients to the data center via Global Protect?
03-27-2023 10:08 AM
Palo has performance tuning suggestions "Performance Tuning of the VM-Series for ESXi".
Discuss with VMware team which of those settings are reasonable to implement.
03-24-2023 01:11 PM
Whenever you see this issue on a VM-series one of the first things that I'm suspect of is the host. Does the VM-300 and the SQL servers reside on the same host? Do you have multiple hosts all running on a shared enclosure? What did the enclosure/host uplink interfaces report when you were seeing the issue?
03-24-2023 02:35 PM
The SQL server and VM PAN are on separate hosts. CPU of the host and guest were both calm at 45%. There are other guests on the (Cisco UCS) host. But as I say there's no spiking during the problem period observed. VM latency and IOPs normal. The uplinks are at least 2x the consumed capacity at their narrowest point. The latency reported by the GP user was just ping latency into the data center.
03-24-2023 05:37 PM
45% as average might seem ok but in virtual environment you also need to take into account that hypervisor has way easier to find available slots for VMs with lower vCPU count than ones that have more vCPUs.
How many vCPUs your VM-300 has?
How many other VMs on same host have?
Does VMware CPU Ready graph of VM-300 go up when users download files? If it does it means you don't have enough physical CPU resources to satisfy need and you need to reserve CPU for VM-300 to perform as expected.
CPU Ready shows time when VM is waiting for CPU resources but hypervisor don't have any to give.
03-27-2023 09:15 AM
The VM has two CPUs. CPU 2 peaks at around 67% and the CPU 1 was 30%. This is out of Solar Winds. "CPU Peak Trend" says the VM is at 70%. There are 30 VMs on the host. I'm not familiar with the CPU ready metric. I'll have to ask the VMW admins perhaps.
Would it be advisable to add CPU to the VM PAN and does it require downtime?
03-27-2023 10:01 AM
Never overprovision VMs.
As I mentioned earlier it is way harder for hypervisor to find timeslots to provide VM physical CPU resources if VM has big amount of vCPUs or has more vCPUs than other VMs have on same host.
So by going from 2vCPU to 4 you might make things worse.
Investigate CPU Ready and analyze with VMware team if it is in accepted treshold.
03-27-2023 10:08 AM
Palo has performance tuning suggestions "Performance Tuning of the VM-Series for ESXi".
Discuss with VMware team which of those settings are reasonable to implement.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
