session end reason resources-unavailable for all traffic

[miaocongcong]

Hi,Guys

The customer's network recently experienced an outage, and  found all the session  end reason  was resources-unavailable ;

I exec the comand " debug dataplane pool statistics" and found there is a parameter in the software pool called Regex Results that has been exhausted. Compared with normal firewalls, this value should not be reduced. I also tried to execute “set deviceconfig setting session resource-limit-behavior bypass” , but this solution is not acceptable to customers.
At present, the customer's system version is 9.1.13 of the PA3000 series, which has been rolled back to 9.1.12, and the parameters are normal
Does anyone know if this is a bug?

Customer needs an official explanation，thanks in advance！

[PavelK]

Thank you for the post @miaocongcong

I experienced exactly the same issue in mid of February after I upgraded PA-3220 to 9.1.13. The command you executed: "debug dataplane pool statistics" is the right way to drill down the issue. As an immediate measure we failed over the Firewall and rebooted the one that was having an issue and opened a TAC case. After spending some time on the call, TAC advised to downgrade the Firewall that solved the issue, however we could not get anything official regarding bug. Based on the input from TAC, in version 9.1.13 there were some issues that engineering team is looking into, but there is no official bug id for this.

Kind Regards

Pavel

[upelister]

Hello with 9.1.13-h3 we hit this bug, we are waiting TAC response. Downgrade means SSL loop vulnerability. 

https://security.paloaltonetworks.com/CVE-2022-0778

[upelister]

Do you have any idea why "set deviceconfig setting session resource-limit-behavior bypass" solution is not acceptable to customers?

[ValleyComm]

Because when you bypass, you are bypassing L7 threat protection. 

[PavelK]

Today, PAN-OS 9.1.14 was released. This issue was addressed in this release: PAN-189468

https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-release-notes/pan-os-9-1-addressed-issues/pan-os...

Kind Regards

Pavel

[bvandi · Accepted solution]

9.1.14 does not appear to have successfully addressed this issue.  We upgraded our 5220s over the weekend and are now facing the same issue.

admin@Firewall(active)> debug dataplane pool statistics | match Regex

[12] Regex Results             (13272):     1020/8192     0x800000042bfda380 205

[PavelK]

Thank you for feedback and sorry to hear that @bvandi

to be honest, there is not much I can add to this. Myself, I am no longer facing this issue and can't comment on this anymore. At this point I would open a ticket to TAC to address this issue.

Kind Regards

Pavel

[William-Wu]

From the Article: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000004ODICA2&lang=en_US%E2%80%A...

Symptom
After upgrading PAN-OS to 9.1.13 or 10.0.10, unexpected traffic failure may occurs and traffic log shows the session end reason "resources-unavailable".

Environment

• All platforms including VM firewalls
• Firewalls running on PAN-OS 9.1.13 or 10.0.10 (not affected to other PAN-OS versions)

Cause
If you can see the session end reason "resources-unavailable" under traffic log without resource usage spike after upgrading PAN-OS to affected versions, please check whether the counter "aho_alloc_lookup_failed" is increasing or not.

> show counter global name aho_alloc_lookup_failed

Name:           aho_alloc_lookup_failed
Value:          184328 <<<<<<---------------!!!!!!
Severity:       Warning
Category:       aho
Aspect:         resource
Description:    failed to alloc regex lookup

If you can see the above counter is increasing, please also check whether the "Regex Results" is depleted.

> debug dataplane pool statistics | match "Regex Results"
[12] Regex Results             (13272):     15/8192     0x8000000109219180

If your firewall matches to the above symptoms, the issue might be hitting to PAN-189468.
Please open a new support case via CSP with uploading tech-support file, if you need our assistance for checking the condition.

Resolution

• This issue was fixed in PAN-OS 9.1.14 and 10.0.10-h1. Please upgrade PAN-OS if you are using affected versions (PAN-189468 only affects on 9.1.13 or 10.0.10).
• To release "Regex Result" memory pool on affected PAN-OS, you have to reboot the firewall.

[upelister]

Hello in 9.1.14-h1 version this bug is fixed. After upgrade no issue seen.

Have a nice day.

[miaocongcong]

Thx for your reply，got it ！

[bvandi]

I'm not seeing anything in the 9.1.14-h1 release notes which would indicate this issue has been addressed.  May I ask what PAN-OS version you upgraded from?

[ValleyComm]

Its PAN issue ID PAN-189468 and it was fixed in 9.1.14

[bvandi]

As stated previously in this thread, 9.1.14 did not address the resource issue for us....hence my reason for asking.

[agil.puthukudynellikka]

show counter global name aho_alloc_lookup_failed always shows 0 even in 9.1.13 when the issue reported . & debug dataplane pool statistics | match "Regex Results" shows 8192/8192 on 9.1.13 -h3 , 9.1.14 h1 on 5220 . this indicate a similar unknown bug in 9.1.14 -h1 .