topic Re: Latency on Internal Interface in General Topics

Latency on Internal Interface

Farzana — Mon, 08 Jan 2018 03:34:02 GMT

Hello,

Using PAN-OS 8.0.7. When we ping a trusted interface, we see latency up and down. Any clues?

root@test-machine:~# ping 10.2.2.100
PING 10.2.2.100 (10.2.2.100) 56(84) bytes of data.
64 bytes from 10.2.2.100: icmp_seq=1 ttl=63 time=3.46 ms
64 bytes from 10.2.2.100: icmp_seq=2 ttl=63 time=1.25 ms
64 bytes from 10.2.2.100: icmp_seq=3 ttl=63 time=19.9 ms
64 bytes from 10.2.2.100: icmp_seq=4 ttl=63 time=17.9 ms
64 bytes from 10.2.2.100: icmp_seq=5 ttl=63 time=18.6 ms
64 bytes from 10.2.2.100: icmp_seq=6 ttl=63 time=16.6 ms
64 bytes from 10.2.2.100: icmp_seq=7 ttl=63 time=3.39 ms
64 bytes from 10.2.2.100: icmp_seq=8 ttl=63 time=2.25 ms
64 bytes from 10.2.2.100: icmp_seq=9 ttl=63 time=11.5 ms
64 bytes from 10.2.2.100: icmp_seq=10 ttl=63 time=17.9 ms
64 bytes from 10.2.2.100: icmp_seq=11 ttl=63 time=9.48 ms
64 bytes from 10.2.2.100: icmp_seq=12 ttl=63 time=11.3 ms
64 bytes from 10.2.2.100: icmp_seq=13 ttl=63 time=7.23 ms
64 bytes from 10.2.2.100: icmp_seq=14 ttl=63 time=14.1 ms
64 bytes from 10.2.2.100: icmp_seq=15 ttl=63 time=3.14 ms
64 bytes from 10.2.2.100: icmp_seq=16 ttl=63 time=1.15 ms
64 bytes from 10.2.2.100: icmp_seq=17 ttl=63 time=9.94 ms
64 bytes from 10.2.2.100: icmp_seq=18 ttl=63 time=18.0 ms
64 bytes from 10.2.2.100: icmp_seq=19 ttl=63 time=16.8 ms
64 bytes from 10.2.2.100: icmp_seq=20 ttl=63 time=14.9 ms
64 bytes from 10.2.2.100: icmp_seq=21 ttl=63 time=12.5 ms
64 bytes from 10.2.2.100: icmp_seq=22 ttl=63 time=4.19 ms
64 bytes from 10.2.2.100: icmp_seq=23 ttl=63 time=1.17 ms
64 bytes from 10.2.2.100: icmp_seq=24 ttl=63 time=11.1 ms
64 bytes from 10.2.2.100: icmp_seq=25 ttl=63 time=10.6 ms
64 bytes from 10.2.2.100: icmp_seq=26 ttl=63 time=8.15 ms
64 bytes from 10.2.2.100: icmp_seq=27 ttl=63 time=7.21 ms
64 bytes from 10.2.2.100: icmp_seq=28 ttl=63 time=6.46 ms
64 bytes from 10.2.2.100: icmp_seq=29 ttl=63 time=2.64 ms

Re: Latency on Internal Interface

reaper — Mon, 08 Jan 2018 09:07:37 GMT

hi @Farzana!

Are you experiencing slowdowns in packets passing through the firewall, is there packetloss or lots of resends?

What's the current load on the dataplane?

Pings to the dataplane interface get the lowest possible priority when processing packets, so if your dataplane is under any type of load you may see some latency

Is there a direct link from your host to the firewall, is there a routing device or a switch in between?

did you verify the speed and duplex settings of the firewall interface to match what it is connected to?

Re: Latency on Internal Interface

Retired Member — Mon, 08 Jan 2018 18:58:50 GMT

Same "issue" here.

This latency when ping the interface adress started after upgrade to PAN 8.0.x

previous versions had only higher latency when the firewall or interace was under high load,

The mgmt interface does not show latency,

Througput and latency when going "through" the device is normal and the same as previous pan-os versions

Also when dataplane usage is 0 the latency is bouncing between 2 and 12 ms

Re: Latency on Internal Interface

BPry — Mon, 08 Jan 2018 18:42:04 GMT

@Farzana,

Just wanted to state that I can't reproduce the issue on either 3020s, 3050s, or 200s all running 8.0.7. Do you have a physical firewall or is this through a VM model?

Re: Latency on Internal Interface

Retired Member — Mon, 08 Jan 2018 18:47:42 GMT

@BPryFor me its a VM-50 stand alone and VM-100 HA clusters.

All our physical firewall's are still on 7.1.1x

Latency for the interfaces IP's happends on all 8.0.x versions

Re: Latency on Internal Interface

BPry — Mon, 08 Jan 2018 18:48:55 GMT

@Retired Member,

Interesting. Maybe this is actually limited to the VMs and doesn't actually effect hardware units?

Re: Latency on Internal Interface

Retired Member — Mon, 08 Jan 2018 18:57:59 GMT

@BPry I think you are right, that is only affect the VM models

Re: Latency on Internal Interface

Farzana — Wed, 10 Jan 2018 22:00:08 GMT

Hi,

Just wanted to let you know I have logged a case with TAC regarding the issue.

So far we have seen that when DPDK is off, latency is dropped. Still checking the issue.

I will update it here.

Re: Latency on Internal Interface

Retired Member — Wed, 07 Feb 2018 17:20:49 GMT

@Farzana Any news on this?

Re: Latency on Internal Interface

Farzana — Wed, 07 Feb 2018 21:24:53 GMT

Hi @Retired Member,

Thank you for the follow up. TAC asked to perform the following.

1. Perform ping test and take specific packet capture by filtering only source and destination in both directions for 15 minutes (Kindly note down the time).

2. Capture output of ethernet1 Interface from ESXi Host at start and end of testing after 15 minutes.
net-stats -l | grep <Firewall name>

Sample output below

ID Number DVS Port No MAC-Address Interface
33554978 5 9 DvsPortset-0 00:49:3b:ee:e3:14 FW.eth5

The above output would give port details for the Firewall.

Once port detail is obtained, for eth1 interface run below command:
cat /net/portsets/<DVS Port No>/ports/<ID Number of eth1>/vmxnet3/rxSummary

3. Take output of below command after every 5 minutes:

debug dataplane pow performance

However, after trying the following command latency has dropped to expectancy level. So our customer did not go ahead with further testing. Case is closed.

admin@PA-VM> show system setting dpdk-pkt-io

Device current Packet IO mode: DPDK
Device DPDK Packet IO capable: yes
Device default Packet IO mode: DPDK

admin@PA-VM> set system setting dpdk-pkt-io off
Enabling/disabling DPDK Packet IO mode requires a device reboot. Do you want to continue? (y or n)

Device is now in non-DPDK IO mode, please reboot device
admin@PA-VM>