topic Software packet buffer depletion in General Topics

Software packet buffer depletion

Dulle — Fri, 12 Feb 2016 10:21:37 GMT

We're currently observing something quite interesting:
On our highly oversized PA-5050 firewall, software packet buffer 0 is, for several hours a day exhausted.

This is the platform (pair that runs in High Avalailability A/P):
family: 5000
model: PA-5050
sw-version: 7.0.4

This is the anomaly:

> debug dataplane pool statistics

Software Pools
[ 0] software packet buffer 0 ( 512): 1/32768 0x8000000020c00680
[ 1] software packet buffer 1 ( 1024): 23178/32768 0x8000000021c20780
[ 2] software packet buffer 2 ( 2048): 31775/32768 0x8000000023c40880
[ 3] software packet buffer 3 (33280): 24528/24576 0x8000000027c60980
[ 4] software packet buffer 4 (66048): 304/304 0x8000000058878a80
|
|
[17] FPTCP segs ( 16): 6703/49152 0x80000000d8f68a80

The load on the firewall is minimal:

> show session info
--------------------------------------------------------------------------------
Number of sessions supported: 2000000
Number of active sessions: 38996
Number of active TCP sessions: 29985
Number of active UDP sessions: 8434
Number of active ICMP sessions: 273
Number of active BCAST sessions: 0
Number of active MCAST sessions: 0
Number of active predict sessions: 774
Session table utilization: 1%
Number of sessions created since bootup: 355140861
Packet rate: 16892/s
Throughput: 54252 kbps
New connection establish rate: 760 cps
--------------------------------------------------------------------------------

Anyone else seeing something similar ?

(and yes, case has been opened with PAN-support)

Re: Software packet buffer depletion

bmorris1 — Fri, 12 Feb 2016 16:58:58 GMT

Hi Dulle,

Seems odd. What are the dataplane usage %s? Are they normal or high? Any particular process that is high?

> show running resource-monitor

Might be worth rebooting the firewall/data plane to clear these out. Also worth an upgrade to the latest minor release but I suspect support can give you better assistance in findout out the root cause.

regards,

Ben

Re: Software packet buffer depletion

Dulle — Mon, 15 Feb 2016 10:58:31 GMT

Update:

Research by PAN-support revealed that this depletion in our case, is related to traffic with destination address 173.255.112.173 (stream.pushbullet.com/).
The application is classsified in the session table as 'pushbullet', then changes to 'websocket' and finishes off as 'web-browsing' when logged.

The sessions are persistent throug the firewall even when the client is disconnected.

Even if the number of sessions to stream.pushbullet.com are modest, it seems somehow to bleed out the 'software packet buffer 0'
If this occcurs,it did help in our case to run a 'clear session all filter destination 173.255.112.173'
Also if the depletion causes problems (if all software packet buffer 0-4 are depleted, possibly dataplane reset), create firewall rule denying traffic to 173.255.112.173.

(Thanks a lot to PAN-support engineer Nikola for invaluable help)

Re: Software packet buffer depletion

Dulle — Fri, 11 Mar 2016 08:54:20 GMT

Some more information for those who might care:

The problem is not resolved but it is apparent that it's related to the App 'Pushbullet' ( see www.pushbullet.com)

This is the result of 3(!) users in our company running Pushbullet as an extention i Chrome browser (still on the PA-5050)

FW(active)> debug dataplane pool statistics

DP dp0:

Software Pools

[ 0] software packet buffer 0 ( 512): 1/32768 0x8000000020c00680

[ 1] software packet buffer 1 ( 1024): 17781/32768 0x8000000021c20780

[ 2] software packet buffer 2 ( 2048): 31141/32768 0x8000000023c40880

[ 3] software packet buffer 3 (33280): 24506/24576 0x8000000027c60980

[ 4] software packet buffer 4 (66048): 304/304 0x8000000058878a80

[17] FPTCP segs ( 16): 1/49152 0x80000000d8f68a80

We play a game we call 'software buffer chicken' and see if the FPTCP pool depletion can cause a dp0 reset.
The firewall chickes out, and seems to reset its pools. We win 😄

Also worth noting: the problem occurs only if you run a decryption policy that the traffic hits.
(Hence the Forward Proxy TCP segs pool depeltion)

(We now do a no_decrypt for traffiic to IP 173.255.112.173, the home of Pushbullet, to avoid trouble over the weekend....)

Re: Software packet buffer depletion

Lucky — Fri, 11 Mar 2016 09:02:49 GMT

Thanks for follow up. Please tell me that you already updated to 7.0.5-h2, I see 7.0.4 in the first post?

BR

Luciano

Re: Software packet buffer depletion

Dulle — Fri, 11 Mar 2016 10:44:35 GMT

Nah... Don't like the hotfix solutions, still 7.0.4.

And since neither 7.0.5 or 7.0.5-h2 has any description of a fix for this particular...anomaly, we haven't bothered.

Re: Software packet buffer depletion

Lucky — Sat, 12 Mar 2016 07:41:35 GMT

Hi Dulle,

If I may offer my opinion (IMHO): hotfix or not hotfix, this is regular update (it is called hotfix because it just patched up a thing or two on 7.0.5 that was already ready to go out). It does fix important issues that weren't described in much detail in release notes so that users that did not have a chance to update yet aren't fully exposed. Describing how you fixed the issue that isn't publicly known is particularly touchy subject, isn't it? If you are taking care of thousands and thousands of users as PAN - if this was academic discussion by all means I would like to see full details, but since we are dealing with real world and trying to protect people in real time - I can understand PAN reasons to disclose as little as possible at this time but publish patches. That being said, if vendor says something is critical, I patch (iPatch?), even my own phone 🙂

Best regards