A weird high latency ping issue with PaloAlto running version 8.1.15 hotfix 3

[dtran]

I have this really weird issue and I don't know how to solve it. I noticed it when I first migrating from Checkpoint over to PaloAlto while doing verification in my Development environment.

- A PAN 5050 running 8.1.15 hotfix 3.  I have a CentOS host (LinuxB) behind the PAN and another CentOS host (LinuxA) outside the PAN.  FW rule is any any allow between these two host and the rule is at the top of the rulebase and nothing else.  The PAN firewall is managed by Panorama,  The interface on the PAN is VLAN tagging on a single physical interface,

- I do a 100 ping from LinuxA to LinuxB, the average RTT is 1.4ms.  The Linux machines and PAN are connected to the same L2 Cisco 3750 switch.  My background prior to PAN is Cisco so I am very familiar with Cisco products. 

- I took out the PAN firewall and put in either a Cisco ASA or IOS router, using the same cable and switchport that I use on the PAN firewall.  When I do 100 ping from LinuxA to LinuxB, the average RTT is 0.9ms, almost a 50% improvement in latency.

I go back and check the latency in my production environment regarding ping (we use open source smokeping tool to monitor our network/firewall devices) before the cutover from Checkpoint to PAN and I also notice the same thing. 

I want to upgrade the PAN 5050 to 8.1.17 and test again; unfortunately, we ran out of support on the device so I could no longer download 8.1.17 for the 5050 but I don't think it would have made any differences.

Thoughts?

[MP18]

@dtran 

Does your ping rule has security profiles attached to it?

Other thing i can say is for application ping do a test without any security profile to see if it make any difference.

Also try ping from PA CLI by using source IP and see any difference.

Regards

[dtran]

"

Does your ping rule has security profiles attached to it?

Other thing i can say is for application ping do a test without any security profile to see if it make any difference."

No ping rule does NOT have any security profiles attached to it.

[Broderick2352]

How did you check the connectivity between the mgmt interface and the internet? 

[dtran]

[dtran]

as you can see in the graph, the high latency is ping across PAN firewall while low latency is across non PAN firewall.  The endpoints are the same, L2 switchport is the same.  The only difference is the firewall.

[BPry]

@dtran,

ICMP traffic is never fast tracked, and while a 50% increase sounds bad your talking about roughly a .5ms difference. What happens if you actually try testing with legitimate traffic (websites, DB calls, ect)? 

[dtran]

50% difference does sound bad.

[Daniel_Garry]

Hi ,

Did you ever get an answer for this? We have a similar issue, I swapped out SW FW for Palo and the pings went from >1ms to 2/3 ms even the odd 16ms.

All traffic is one the LAN same intrazone and has no profiles associated. We are seeing people having issues now with applications etc

[dtran]

@Daniel_Garry:  The response I received back from PAN support, after wasting so many hours with them, was that "it is expected".  To me, that is NOT an acceptable response, at the same time, I didn't want to waste anymore of my time to fight PAN support any further.  The support from TAC was not good in this situation. 

[Daniel_Garry]

Great thanks. I have a case opened with them now also, but feel like they won't give a good answer

[dtran]

No they will not.  If anything else, they will waste your time and resource.

[OtakarKlier]

Hello All,

Hopefully you are running a newer version of PANOS as t8.x is very old. Please check the following for newer version recommendations: 

https://live.paloaltonetworks.com/t5/customer-resources/support-pan-os-software-release-guidance/ta-...

Regards,

[dtran]

FYI:  I am running version 10.1.6-h6 and I am still seeing the ping latency issues. 

[OtakarKlier]

Hello,

Check out this article and see if it helps:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000HBjNCAW

Regards,