Slow VPN throughput after update to 5.0.7

[lanhan2010]

Hello Community

Has anyone recognized some performance issues (extremely slow ip-sec throughput, hanging sessions) after updating to 5.0.7

A Firewall reboot solves this issue, but it's coming up again after some days.

Many thanks

Hannes

[HULK]

Hello Hannes,

Could you please apply below mentioned command after every 10 minutes and share the o/p with us.

> debug data-plane pool statistics

> show system statistics session

> show running resource-monitor

Thanks

[hsnetworks01]

Hello.

We have same performance after update from 5.0.5 so no performance issue.

Check on PA-2050, PA-500 and PA-200

[mbutt]

Please run the following commands and show the entire output. The sample out would seem something like below and more.

admin@500> debug dataplane pool statistics

Verify Software pools are not depleted:

Software Pools

[ 0] software packet buffer 0  : 16384/16384    0x8000000021800680

[ 1] software packet buffer 1  : 8192/8192     0x8000000022010700

[ 2] software packet buffer 2  : 8192/8192     0x8000000022818780

[ 3] software packet buffer 3  : 4096/4096     0x8000000023820800

When the issue is happening If by any chance the number is reaching to 1 that would indicate that some buffer is leaking. If this is the case you will need to open a case with Tech support to further investigate the issue.

Hope this helps.

Thanks

[CRHC]

Upgraded our 3050 pair 2 weeks ago and have not heard of any issues with VPN yet.  Will keep monitoring now.  Resources appear in good shape.  Thanks!

[lanhan2010]

Thanks for your input. I want to execute the commands and post the result.

Btw, the issue happend again on PA-200 and PA-500.

Currently everything is working fine: Output from PA-200

(active)> debug dataplane pool statistics

Hardware Pools

[ 0] Packet Buffers            :    57218/57344    0x80000000b2000000

[ 1] Work Queue Entries        :   229310/229376   0x80000000b9000000

[ 2] Output Buffers            :     1012/1024     0x8000000000103000

[ 3] DFA Result                :     2048/2048     0x8000000000203000

     DFA Result                :

[ 4] Timer Buffers             :     4092/4096     0x8000000000403000

     Timer Buffers             :

[ 5] PAN_FPA_LWM_POOL          :     1024/1024     0x8000000000803000

[ 6] PAN_FPA_ZIP_POOL          :     1023/1024     0x8000000000843000

[ 7] PAN_FPA_BLAST_POOL        :       64/64       0x8000000000a48000

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x80000000bac00680

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700

[ 2] software packet buffer 2  :    10726/16384    0x80000000bbc18780

[ 3] software packet buffer 3  :     4096/4096     0x80000000bdc28800

[ 4] software packet buffer 4  :      304/304      0x80000000c5e2c880

[ 5] ZIP Results               :     1024/1024     0x8000000000cb0948

[ 6] CTD Flow                  :    64979/65536    0x80000000d2ebb080

[ 7] CTD AV Block              :       32/32       0x80000000e49a20f0

[ 8] SML VM Fields             :    69561/69632    0x80000000e49aa1f0

[ 9] SML VM Vchecks            :    32768/32768    0x80000000e4c0e270

[10] Detector Threats          :    65508/65536    0x80000000e4cae2f0

[11] CTD DLP FLOW              :    16380/16384    0x80000000e5bb9998

[12] CTD DLP DATA              :     1024/1024     0x80000000e5dc9a18

[13] CTD DECODE FILTER         :    32768/32768    0x80000000e5ecaaa0

[14] Regex Results             :     2048/2048     0x80000000e608b088

[15] TIMER Chunk               :   131072/131072   0x80000000ee5b7ad8

[16] FPTCP segs                :    32768/32768    0x80000000f0637b58

[17] Proxy session             :     1024/1024     0x80000000f06d7bd8

[18] SSL Handshake State       :     1024/1024     0x80000000f0732c58

[19] SSL State                 :     2048/2048     0x80000000f08cdcd8

[20] SSH Handshake State       :       16/16       0x80000000f093fd58

[21] SSH State                 :      128/128      0x80000000f095b498

[22] TCP host connections      :       15/16       0x80000000f09aa478

*********************************

show system statistics session

System Statistics: ('q' to quit, 'h' for help)

Device is up          : 4 days 11 hours 10 mins 18 sec

Packet rate           : 148/s

Throughput            : 1030 Kbps

Total active sessions : 572

Active TCP sessions   : 93

Active UDP sessions   : 477

Active ICMP sessions  : 0

**********************************

[HULK]

Hello Ianhan,

I think this a buffer leak issue on PAN _OS  5.0.7, pls see below o/p.

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x80000000bac00680

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700 >>>>>>>>>>>>>>>>>>>>>>>>>no buffer available.

[ 2] software packet buffer 2  :    10726/16384    0x80000000bbc18780

[ 3] software packet buffer 3  :     4096/4096     0x80000000bdc28800

[ 4] software packet buffer 4  :      304/304      0x80000000c5e2c880

This symptom can cause slow performance issues through this PAN firewall. I will suggest you to open a case with PAN support, so that we will give you a temp fix version or  you can stay back with your previous PAN-OS version.

Hope this helps.

Thanks

[lanhan2010]

Hi Hulk

Thanks for your fast response. Just fo my correct understanding:

The value for the software packet buffer should be as high as possible?

Directly after the firewall reboot I can see the following value:

[ 1] software packet buffer 1  :     8134/8192     0x80000000bb410700

Some hours later the value changed to:

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700

Many thanks

[apasupulati]

The values represent available/total buffers. 1/#### means that there are no available buffers and dataplane pools have been exhausted. Standard behavior for the software buffers is for them to dynamically allocate as needed and dynamically recover when no longer in use.

[HULK · Accepted solution]

Hello Ianhan,

As Apasupulati explain about the buffer functionality in PAN-OS very well on above post. So, once the buffer will exhaust ( 1/8192     0x80000000bb410700 ), all incoming/outgoing packet will be in queue for a long time to allocate packet buffer for further processing. As a result you will see an unexpected delay / slowness of traffic processing. Every time you will reboot this firewall, the packet buffer counter will reset to normal ( 8134/8192     0x80000000bb410700).

The PAN engineering team has already identified the issue with this version PAN-5.0.7 and fixed in a temp-fix version. For better and longer stability, i would recommend you to upgrade the PAN-OS of this firewall.

Hope this helps.

Thanks

[shasnain]

The Software buffer leak issue has been fixed in 5.0.7-h2. You will have to open a case with support for the software to be available for you to download and install.

Thanks,

Syed R hasnain

[ericgearhart]

Again, how did this make it past QA is my big question.

[lanhan2010]

I forwared the information now to our external partner, they must create a Case regarding this issue. Hopefully weare able to download the "fixed release" from PaloALto soon.

I'll keep you up to date. Thanks again for all your explanations

[gafrol]

Had the same issue a week ago. Downgraded to 5.0.6 since hotfix was not available at that time. All working fine with 5.0.6.

[lanhan2010]

Final Info from Tech support:

From CLI: admin@device# set deviceconfig setting pow wqe-swbuf-ref no

admin@device# commit

- or -

5.0.7-h2 hotfix (tech support must enable the download of this version in the software center)