Slow VPN throughput after update to 5.0.7

Reply
Highlighted
L1 Bithead

Slow VPN throughput after update to 5.0.7

Hello Community

Has anyone recognized some performance issues (extremely slow ip-sec throughput, hanging sessions) after updating to 5.0.7

A Firewall reboot solves this issue, but it's coming up again after some days.

Many thanks

Hannes

Tags (2)

Accepted Solutions
Highlighted
L7 Applicator

Re: Slow VPN throughput after update to 5.0.7

Hello Ianhan,


As Apasupulati explain about the buffer functionality in PAN-OS very well on above post. So, once the buffer will exhaust ( 1/8192     0x80000000bb410700 ), all incoming/outgoing packet will be in queue for a long time to allocate packet buffer for further processing. As a result you will see an unexpected delay / slowness of traffic processing. Every time you will reboot this firewall, the packet buffer counter will reset to normal ( 8134/8192     0x80000000bb410700).


The PAN engineering team has already identified the issue with this version PAN-5.0.7 and fixed in a temp-fix version. For better and longer stability, i would recommend you to upgrade the PAN-OS of this firewall.


Hope this helps.


Thanks

View solution in original post


All Replies
Highlighted
L7 Applicator

Re: Slow VPN throughput after update to 5.0.7

Hello Hannes,

Could you please apply below mentioned command after every 10 minutes and share the o/p with us.

> debug data-plane pool statistics

> show system statistics session

> show running resource-monitor

Thanks

Highlighted
Not applicable

Re: Slow VPN throughput after update to 5.0.7

Hello.

We have same performance after update from 5.0.5 so no performance issue.

Check on PA-2050, PA-500 and PA-200

Highlighted
L5 Sessionator

Re: Slow VPN throughput after update to 5.0.7

Please run the following commands and show the entire output. The sample out would seem something like below and more.

admin@500> debug dataplane pool statistics

Verify Software pools are not depleted:

Software Pools

[ 0] software packet buffer 0  : 16384/16384    0x8000000021800680

[ 1] software packet buffer 1  : 8192/8192     0x8000000022010700

[ 2] software packet buffer 2  : 8192/8192     0x8000000022818780

[ 3] software packet buffer 3  : 4096/4096     0x8000000023820800

When the issue is happening If by any chance the number is reaching to 1 that would indicate that some buffer is leaking. If this is the case you will need to open a case with Tech support to further investigate the issue.

Hope this helps.

Thanks

Highlighted
L4 Transporter

Re: Slow VPN throughput after update to 5.0.7

Upgraded our 3050 pair 2 weeks ago and have not heard of any issues with VPN yet.  Will keep monitoring now.  Resources appear in good shape.  Thanks!

Highlighted
L1 Bithead

Re: Slow VPN throughput after update to 5.0.7

Thanks for your input. I want to execute the commands and post the result.

Btw, the issue happend again on PA-200 and PA-500.

Currently everything is working fine: Output from PA-200


(active)> debug dataplane pool statistics

Hardware Pools

[ 0] Packet Buffers            :    57218/57344    0x80000000b2000000

[ 1] Work Queue Entries        :   229310/229376   0x80000000b9000000

[ 2] Output Buffers            :     1012/1024     0x8000000000103000

[ 3] DFA Result                :     2048/2048     0x8000000000203000

     DFA Result                :

[ 4] Timer Buffers             :     4092/4096     0x8000000000403000

     Timer Buffers             :

[ 5] PAN_FPA_LWM_POOL          :     1024/1024     0x8000000000803000

[ 6] PAN_FPA_ZIP_POOL          :     1023/1024     0x8000000000843000

[ 7] PAN_FPA_BLAST_POOL        :       64/64       0x8000000000a48000

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x80000000bac00680

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700

[ 2] software packet buffer 2  :    10726/16384    0x80000000bbc18780

[ 3] software packet buffer 3  :     4096/4096     0x80000000bdc28800

[ 4] software packet buffer 4  :      304/304      0x80000000c5e2c880

[ 5] ZIP Results               :     1024/1024     0x8000000000cb0948

[ 6] CTD Flow                  :    64979/65536    0x80000000d2ebb080

[ 7] CTD AV Block              :       32/32       0x80000000e49a20f0

[ 8] SML VM Fields             :    69561/69632    0x80000000e49aa1f0

[ 9] SML VM Vchecks            :    32768/32768    0x80000000e4c0e270

[10] Detector Threats          :    65508/65536    0x80000000e4cae2f0

[11] CTD DLP FLOW              :    16380/16384    0x80000000e5bb9998

[12] CTD DLP DATA              :     1024/1024     0x80000000e5dc9a18

[13] CTD DECODE FILTER         :    32768/32768    0x80000000e5ecaaa0

[14] Regex Results             :     2048/2048     0x80000000e608b088

[15] TIMER Chunk               :   131072/131072   0x80000000ee5b7ad8

[16] FPTCP segs                :    32768/32768    0x80000000f0637b58

[17] Proxy session             :     1024/1024     0x80000000f06d7bd8

[18] SSL Handshake State       :     1024/1024     0x80000000f0732c58

[19] SSL State                 :     2048/2048     0x80000000f08cdcd8

[20] SSH Handshake State       :       16/16       0x80000000f093fd58

[21] SSH State                 :      128/128      0x80000000f095b498

[22] TCP host connections      :       15/16       0x80000000f09aa478

*********************************

show system statistics session

System Statistics: ('q' to quit, 'h' for help)

Device is up          : 4 days 11 hours 10 mins 18 sec

Packet rate           : 148/s

Throughput            : 1030 Kbps

Total active sessions : 572

Active TCP sessions   : 93

Active UDP sessions   : 477

Active ICMP sessions  : 0

**********************************

Highlighted
L7 Applicator

Re: Slow VPN throughput after update to 5.0.7

Hello Ianhan,

I think this a buffer leak issue on PAN _OS  5.0.7, pls see below o/p.

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x80000000bac00680

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700 >>>>>>>>>>>>>>>>>>>>>>>>>no buffer available.

[ 2] software packet buffer 2  :    10726/16384    0x80000000bbc18780

[ 3] software packet buffer 3  :     4096/4096     0x80000000bdc28800

[ 4] software packet buffer 4  :      304/304      0x80000000c5e2c880

This symptom can cause slow performance issues through this PAN firewall. I will suggest you to open a case with PAN support, so that we will give you a temp fix version or  you can stay back with your previous PAN-OS version.


Hope this helps.


Thanks

Highlighted
L1 Bithead

Re: Slow VPN throughput after update to 5.0.7

Hi Hulk

Thanks for your fast response. Just fo my correct understanding:

The value for the software packet buffer should be as high as possible?

Directly after the firewall reboot I can see the following value:

[ 1] software packet buffer 1  :     8134/8192     0x80000000bb410700

Some hours later the value changed to:

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700

Many thanks

Highlighted
L4 Transporter

Re: Slow VPN throughput after update to 5.0.7

The values represent available/total buffers. 1/#### means that there are no available buffers and dataplane pools have been exhausted. Standard behavior for the software buffers is for them to dynamically allocate as needed and dynamically recover when no longer in use.

Highlighted
L7 Applicator

Re: Slow VPN throughput after update to 5.0.7

Hello Ianhan,


As Apasupulati explain about the buffer functionality in PAN-OS very well on above post. So, once the buffer will exhaust ( 1/8192     0x80000000bb410700 ), all incoming/outgoing packet will be in queue for a long time to allocate packet buffer for further processing. As a result you will see an unexpected delay / slowness of traffic processing. Every time you will reboot this firewall, the packet buffer counter will reset to normal ( 8134/8192     0x80000000bb410700).


The PAN engineering team has already identified the issue with this version PAN-5.0.7 and fixed in a temp-fix version. For better and longer stability, i would recommend you to upgrade the PAN-OS of this firewall.


Hope this helps.


Thanks

View solution in original post

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!