topic Re: Slow VPN throughput after update to 5.0.7 in General Topics

Slow VPN throughput after update to 5.0.7

lanhan2010 — Fri, 13 Sep 2013 07:39:47 GMT

Hello Community

Has anyone recognized some performance issues (extremely slow ip-sec throughput, hanging sessions) after updating to 5.0.7

A Firewall reboot solves this issue, but it's coming up again after some days.

Many thanks

Hannes

Re: Slow VPN throughput after update to 5.0.7

HULK — Fri, 13 Sep 2013 17:00:40 GMT

Hello Hannes,

Could you please apply below mentioned command after every 10 minutes and share the o/p with us.

> debug data-plane pool statistics

> show system statistics session

> show running resource-monitor

Thanks

Re: Slow VPN throughput after update to 5.0.7

hsnetworks01 — Fri, 13 Sep 2013 18:00:58 GMT

Hello.

We have same performance after update from 5.0.5 so no performance issue.

Check on PA-2050, PA-500 and PA-200

Re: Slow VPN throughput after update to 5.0.7

mbutt — Sat, 14 Sep 2013 07:05:53 GMT

Please run the following commands and show the entire output. The sample out would seem something like below and more.

admin@500> debug dataplane pool statistics

Verify Software pools are not depleted:

Software Pools

[ 0] software packet buffer 0 : 16384/16384 0x8000000021800680

[ 1] software packet buffer 1 : 8192/8192 0x8000000022010700

[ 2] software packet buffer 2 : 8192/8192 0x8000000022818780

[ 3] software packet buffer 3 : 4096/4096 0x8000000023820800

When the issue is happening If by any chance the number is reaching to 1 that would indicate that some buffer is leaking. If this is the case you will need to open a case with Tech support to further investigate the issue.

Hope this helps.

Thanks

Re: Slow VPN throughput after update to 5.0.7

MGoodnow — Mon, 16 Sep 2013 16:50:15 GMT

Upgraded our 3050 pair 2 weeks ago and have not heard of any issues with VPN yet. Will keep monitoring now. Resources appear in good shape. Thanks!

Re: Slow VPN throughput after update to 5.0.7

lanhan2010 — Wed, 18 Sep 2013 05:17:47 GMT

Thanks for your input. I want to execute the commands and post the result.

Btw, the issue happend again on PA-200 and PA-500.

Currently everything is working fine: Output from PA-200

(active)> debug dataplane pool statistics

Hardware Pools

[ 0] Packet Buffers : 57218/57344 0x80000000b2000000

[ 1] Work Queue Entries : 229310/229376 0x80000000b9000000

[ 2] Output Buffers : 1012/1024 0x8000000000103000

[ 3] DFA Result : 2048/2048 0x8000000000203000

DFA Result :

[ 4] Timer Buffers : 4092/4096 0x8000000000403000

Timer Buffers :

[ 5] PAN_FPA_LWM_POOL : 1024/1024 0x8000000000803000

[ 6] PAN_FPA_ZIP_POOL : 1023/1024 0x8000000000843000

[ 7] PAN_FPA_BLAST_POOL : 64/64 0x8000000000a48000

Software Pools

[ 0] software packet buffer 0 : 16383/16384 0x80000000bac00680

[ 1] software packet buffer 1 : 1/8192 0x80000000bb410700

[ 2] software packet buffer 2 : 10726/16384 0x80000000bbc18780

[ 3] software packet buffer 3 : 4096/4096 0x80000000bdc28800

[ 4] software packet buffer 4 : 304/304 0x80000000c5e2c880

[ 5] ZIP Results : 1024/1024 0x8000000000cb0948

[ 6] CTD Flow : 64979/65536 0x80000000d2ebb080

[ 7] CTD AV Block : 32/32 0x80000000e49a20f0

[ 8] SML VM Fields : 69561/69632 0x80000000e49aa1f0

[ 9] SML VM Vchecks : 32768/32768 0x80000000e4c0e270

[10] Detector Threats : 65508/65536 0x80000000e4cae2f0

[11] CTD DLP FLOW : 16380/16384 0x80000000e5bb9998

[12] CTD DLP DATA : 1024/1024 0x80000000e5dc9a18

[13] CTD DECODE FILTER : 32768/32768 0x80000000e5ecaaa0

[14] Regex Results : 2048/2048 0x80000000e608b088

[15] TIMER Chunk : 131072/131072 0x80000000ee5b7ad8

[16] FPTCP segs : 32768/32768 0x80000000f0637b58

[17] Proxy session : 1024/1024 0x80000000f06d7bd8

[18] SSL Handshake State : 1024/1024 0x80000000f0732c58

[19] SSL State : 2048/2048 0x80000000f08cdcd8

[20] SSH Handshake State : 16/16 0x80000000f093fd58

[21] SSH State : 128/128 0x80000000f095b498

[22] TCP host connections : 15/16 0x80000000f09aa478

*********************************

show system statistics session

System Statistics: ('q' to quit, 'h' for help)

Device is up : 4 days 11 hours 10 mins 18 sec

Packet rate : 148/s

Throughput : 1030 Kbps

Total active sessions : 572

Active TCP sessions : 93

Active UDP sessions : 477

Active ICMP sessions : 0

**********************************

Re: Slow VPN throughput after update to 5.0.7

HULK — Wed, 18 Sep 2013 07:17:15 GMT

Hello Ianhan,

I think this a buffer leak issue on PAN _OS 5.0.7, pls see below o/p.

Software Pools

[ 0] software packet buffer 0 : 16383/16384 0x80000000bac00680

[ 1] software packet buffer 1 : 1/8192 0x80000000bb410700 >>>>>>>>>>>>>>>>>>>>>>>>>no buffer available.

[ 2] software packet buffer 2 : 10726/16384 0x80000000bbc18780

[ 3] software packet buffer 3 : 4096/4096 0x80000000bdc28800

[ 4] software packet buffer 4 : 304/304 0x80000000c5e2c880

This symptom can cause slow performance issues through this PAN firewall. I will suggest you to open a case with PAN support, so that we will give you a temp fix version or you can stay back with your previous PAN-OS version.

Hope this helps.

Thanks

Re: Slow VPN throughput after update to 5.0.7

lanhan2010 — Wed, 18 Sep 2013 07:57:42 GMT

Hi Hulk

Thanks for your fast response. Just fo my correct understanding:

The value for the software packet buffer should be as high as possible?

Directly after the firewall reboot I can see the following value:

[ 1] software packet buffer 1 : 8134/8192 0x80000000bb410700

Some hours later the value changed to:

[ 1] software packet buffer 1 : 1/8192 0x80000000bb410700

Many thanks

Re: Slow VPN throughput after update to 5.0.7

apasupulati — Wed, 18 Sep 2013 15:18:04 GMT

The values represent available/total buffers. 1/#### means that there are no available buffers and dataplane pools have been exhausted. Standard behavior for the software buffers is for them to dynamically allocate as needed and dynamically recover when no longer in use.

Re: Slow VPN throughput after update to 5.0.7

HULK — Wed, 18 Sep 2013 15:46:59 GMT

Hello Ianhan,

As Apasupulati explain about the buffer functionality in PAN-OS very well on above post. So, once the buffer will exhaust ( 1/8192 0x80000000bb410700 ), all incoming/outgoing packet will be in queue for a long time to allocate packet buffer for further processing. As a result you will see an unexpected delay / slowness of traffic processing. Every time you will reboot this firewall, the packet buffer counter will reset to normal ( 8134/8192 0x80000000bb410700).

The PAN engineering team has already identified the issue with this version PAN-5.0.7 and fixed in a temp-fix version. For better and longer stability, i would recommend you to upgrade the PAN-OS of this firewall.

Hope this helps.

Thanks

Re: Slow VPN throughput after update to 5.0.7

shasnain — Wed, 18 Sep 2013 16:35:18 GMT

The Software buffer leak issue has been fixed in 5.0.7-h2. You will have to open a case with support for the software to be available for you to download and install.

Thanks,

Syed R hasnain

Re: Slow VPN throughput after update to 5.0.7

ericgearhart — Wed, 18 Sep 2013 18:41:17 GMT

Again, how did this make it past QA is my big question.

Re: Slow VPN throughput after update to 5.0.7

lanhan2010 — Thu, 19 Sep 2013 05:28:58 GMT

I forwared the information now to our external partner, they must create a Case regarding this issue. Hopefully weare able to download the "fixed release" from PaloALto soon.

I'll keep you up to date. Thanks again for all your explanations

Re: Slow VPN throughput after update to 5.0.7

gafrol — Thu, 19 Sep 2013 07:39:52 GMT

Had the same issue a week ago. Downgraded to 5.0.6 since hotfix was not available at that time. All working fine with 5.0.6.

Re: Slow VPN throughput after update to 5.0.7

lanhan2010 — Mon, 23 Sep 2013 05:13:54 GMT

Final Info from Tech support:

From CLI: admin@device# set deviceconfig setting pow wqe-swbuf-ref no

admin@device# commit

- or -

5.0.7-h2 hotfix (tech support must enable the download of this version in the software center)