Slow VPN throughput after update to 5.0.7

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Slow VPN throughput after update to 5.0.7

L1 Bithead

Hello Community

Has anyone recognized some performance issues (extremely slow ip-sec throughput, hanging sessions) after updating to 5.0.7

A Firewall reboot solves this issue, but it's coming up again after some days.

Many thanks

Hannes

1 accepted solution

Accepted Solutions

Hello Ianhan,


As Apasupulati explain about the buffer functionality in PAN-OS very well on above post. So, once the buffer will exhaust ( 1/8192     0x80000000bb410700 ), all incoming/outgoing packet will be in queue for a long time to allocate packet buffer for further processing. As a result you will see an unexpected delay / slowness of traffic processing. Every time you will reboot this firewall, the packet buffer counter will reset to normal ( 8134/8192     0x80000000bb410700).


The PAN engineering team has already identified the issue with this version PAN-5.0.7 and fixed in a temp-fix version. For better and longer stability, i would recommend you to upgrade the PAN-OS of this firewall.


Hope this helps.


Thanks

View solution in original post

14 REPLIES 14

L7 Applicator

Hello Hannes,

Could you please apply below mentioned command after every 10 minutes and share the o/p with us.

> debug data-plane pool statistics

> show system statistics session

> show running resource-monitor

Thanks

Hello.

We have same performance after update from 5.0.5 so no performance issue.

Check on PA-2050, PA-500 and PA-200

L5 Sessionator

Please run the following commands and show the entire output. The sample out would seem something like below and more.

admin@500> debug dataplane pool statistics

Verify Software pools are not depleted:

Software Pools

[ 0] software packet buffer 0  : 16384/16384    0x8000000021800680

[ 1] software packet buffer 1  : 8192/8192     0x8000000022010700

[ 2] software packet buffer 2  : 8192/8192     0x8000000022818780

[ 3] software packet buffer 3  : 4096/4096     0x8000000023820800

When the issue is happening If by any chance the number is reaching to 1 that would indicate that some buffer is leaking. If this is the case you will need to open a case with Tech support to further investigate the issue.

Hope this helps.

Thanks

L4 Transporter

Upgraded our 3050 pair 2 weeks ago and have not heard of any issues with VPN yet.  Will keep monitoring now.  Resources appear in good shape.  Thanks!

Thanks for your input. I want to execute the commands and post the result.

Btw, the issue happend again on PA-200 and PA-500.

Currently everything is working fine: Output from PA-200


(active)> debug dataplane pool statistics

Hardware Pools

[ 0] Packet Buffers            :    57218/57344    0x80000000b2000000

[ 1] Work Queue Entries        :   229310/229376   0x80000000b9000000

[ 2] Output Buffers            :     1012/1024     0x8000000000103000

[ 3] DFA Result                :     2048/2048     0x8000000000203000

     DFA Result                :

[ 4] Timer Buffers             :     4092/4096     0x8000000000403000

     Timer Buffers             :

[ 5] PAN_FPA_LWM_POOL          :     1024/1024     0x8000000000803000

[ 6] PAN_FPA_ZIP_POOL          :     1023/1024     0x8000000000843000

[ 7] PAN_FPA_BLAST_POOL        :       64/64       0x8000000000a48000

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x80000000bac00680

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700

[ 2] software packet buffer 2  :    10726/16384    0x80000000bbc18780

[ 3] software packet buffer 3  :     4096/4096     0x80000000bdc28800

[ 4] software packet buffer 4  :      304/304      0x80000000c5e2c880

[ 5] ZIP Results               :     1024/1024     0x8000000000cb0948

[ 6] CTD Flow                  :    64979/65536    0x80000000d2ebb080

[ 7] CTD AV Block              :       32/32       0x80000000e49a20f0

[ 8] SML VM Fields             :    69561/69632    0x80000000e49aa1f0

[ 9] SML VM Vchecks            :    32768/32768    0x80000000e4c0e270

[10] Detector Threats          :    65508/65536    0x80000000e4cae2f0

[11] CTD DLP FLOW              :    16380/16384    0x80000000e5bb9998

[12] CTD DLP DATA              :     1024/1024     0x80000000e5dc9a18

[13] CTD DECODE FILTER         :    32768/32768    0x80000000e5ecaaa0

[14] Regex Results             :     2048/2048     0x80000000e608b088

[15] TIMER Chunk               :   131072/131072   0x80000000ee5b7ad8

[16] FPTCP segs                :    32768/32768    0x80000000f0637b58

[17] Proxy session             :     1024/1024     0x80000000f06d7bd8

[18] SSL Handshake State       :     1024/1024     0x80000000f0732c58

[19] SSL State                 :     2048/2048     0x80000000f08cdcd8

[20] SSH Handshake State       :       16/16       0x80000000f093fd58

[21] SSH State                 :      128/128      0x80000000f095b498

[22] TCP host connections      :       15/16       0x80000000f09aa478

*********************************

show system statistics session

System Statistics: ('q' to quit, 'h' for help)

Device is up          : 4 days 11 hours 10 mins 18 sec

Packet rate           : 148/s

Throughput            : 1030 Kbps

Total active sessions : 572

Active TCP sessions   : 93

Active UDP sessions   : 477

Active ICMP sessions  : 0

**********************************

Hello Ianhan,

I think this a buffer leak issue on PAN _OS  5.0.7, pls see below o/p.

Software Pools

[ 0] software packet buffer 0  :    16383/16384    0x80000000bac00680

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700 >>>>>>>>>>>>>>>>>>>>>>>>>no buffer available.

[ 2] software packet buffer 2  :    10726/16384    0x80000000bbc18780

[ 3] software packet buffer 3  :     4096/4096     0x80000000bdc28800

[ 4] software packet buffer 4  :      304/304      0x80000000c5e2c880

This symptom can cause slow performance issues through this PAN firewall. I will suggest you to open a case with PAN support, so that we will give you a temp fix version or  you can stay back with your previous PAN-OS version.


Hope this helps.


Thanks

Hi Hulk

Thanks for your fast response. Just fo my correct understanding:

The value for the software packet buffer should be as high as possible?

Directly after the firewall reboot I can see the following value:

[ 1] software packet buffer 1  :     8134/8192     0x80000000bb410700

Some hours later the value changed to:

[ 1] software packet buffer 1  :        1/8192     0x80000000bb410700

Many thanks

The values represent available/total buffers. 1/#### means that there are no available buffers and dataplane pools have been exhausted. Standard behavior for the software buffers is for them to dynamically allocate as needed and dynamically recover when no longer in use.

Hello Ianhan,


As Apasupulati explain about the buffer functionality in PAN-OS very well on above post. So, once the buffer will exhaust ( 1/8192     0x80000000bb410700 ), all incoming/outgoing packet will be in queue for a long time to allocate packet buffer for further processing. As a result you will see an unexpected delay / slowness of traffic processing. Every time you will reboot this firewall, the packet buffer counter will reset to normal ( 8134/8192     0x80000000bb410700).


The PAN engineering team has already identified the issue with this version PAN-5.0.7 and fixed in a temp-fix version. For better and longer stability, i would recommend you to upgrade the PAN-OS of this firewall.


Hope this helps.


Thanks

L4 Transporter

The Software buffer leak issue has been fixed in 5.0.7-h2. You will have to open a case with support for the software to be available for you to download and install.

Thanks,

Syed R hasnain

L4 Transporter

Again, how did this make it past QA is my big question.

L1 Bithead

I forwared the information now to our external partner, they must create a Case regarding this issue. Hopefully weare able to download the "fixed release" from PaloALto soon.

I'll keep you up to date. Thanks again for all your explanations

Had the same issue a week ago. Downgraded to 5.0.6 since hotfix was not available at that time. All working fine with 5.0.6.

L1 Bithead

Final Info from Tech support:


From CLI: admin@device# set deviceconfig setting pow wqe-swbuf-ref no

admin@device# commit


- or -


5.0.7-h2 hotfix (tech support must enable the download of this version in the software center)

  • 1 accepted solution
  • 6389 Views
  • 14 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!