session end reason resources-unavailable for all traffic

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

session end reason resources-unavailable for all traffic

L1 Bithead

Hi,Guys

The customer's network recently experienced an outage, and  found all the session  end reason  was resources-unavailable ;

I exec the comand " debug dataplane pool statistics" and found there is a parameter in the software pool called Regex Results that has been exhausted. Compared with normal firewalls, this value should not be reduced. I also tried to execute “set deviceconfig setting session resource-limit-behavior bypass” , but this solution is not acceptable to customers.
At present, the customer's system version is 9.1.13 of the PA3000 series, which has been rolled back to 9.1.12, and the parameters are normal
Does anyone know if this is a bug?

Customer needs an official explanation,thanks in advance!

1 accepted solution

Accepted Solutions

9.1.14 does not appear to have successfully addressed this issue.  We upgraded our 5220s over the weekend and are now facing the same issue.

 

admin@Firewall(active)> debug dataplane pool statistics | match Regex

[12] Regex Results             (13272):     1020/8192     0x800000042bfda380 205

View solution in original post

17 REPLIES 17

Cyber Elite
Cyber Elite

Thank you for the post @miaocongcong

 

I experienced exactly the same issue in mid of February after I upgraded PA-3220 to 9.1.13. The command you executed: "debug dataplane pool statistics" is the right way to drill down the issue. As an immediate measure we failed over the Firewall and rebooted the one that was having an issue and opened a TAC case. After spending some time on the call, TAC advised to downgrade the Firewall that solved the issue, however we could not get anything official regarding bug. Based on the input from TAC, in version 9.1.13 there were some issues that engineering team is looking into, but there is no official bug id for this.

 

Kind Regards

Pavel

Help the community: Like helpful comments and mark solutions.

L3 Networker

Hello with 9.1.13-h3 we hit this bug, we are waiting TAC response. Downgrade means SSL loop vulnerability. 

https://security.paloaltonetworks.com/CVE-2022-0778

UP

L3 Networker

Do you have any idea why "set deviceconfig setting session resource-limit-behavior bypass" solution is not acceptable to customers?

UP

L1 Bithead

Because when you bypass, you are bypassing L7 threat protection. 

Cyber Elite
Cyber Elite

Today, PAN-OS 9.1.14 was released. This issue was addressed in this release: PAN-189468

 

https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-release-notes/pan-os-9-1-addressed-issues/pan-os...

 

PavelK_0-1652919150985.png

Kind Regards

Pavel

Help the community: Like helpful comments and mark solutions.

9.1.14 does not appear to have successfully addressed this issue.  We upgraded our 5220s over the weekend and are now facing the same issue.

 

admin@Firewall(active)> debug dataplane pool statistics | match Regex

[12] Regex Results             (13272):     1020/8192     0x800000042bfda380 205

Cyber Elite
Cyber Elite

Thank you for feedback and sorry to hear that @bvandi

 

to be honest, there is not much I can add to this. Myself, I am no longer facing this issue and can't comment on this anymore. At this point I would open a ticket to TAC to address this issue.

 

Kind Regards

Pavel

Help the community: Like helpful comments and mark solutions.

L1 Bithead

From the Article: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000004ODICA2&lang=en_US%E2%80%A...

Symptom
After upgrading PAN-OS to 9.1.13 or 10.0.10, unexpected traffic failure may occurs and traffic log shows the session end reason "resources-unavailable".

Environment

  • All platforms including VM firewalls
  • Firewalls running on PAN-OS 9.1.13 or 10.0.10 (not affected to other PAN-OS versions)



Cause
If you can see the session end reason "resources-unavailable" under traffic log without resource usage spike after upgrading PAN-OS to affected versions, please check whether the counter "aho_alloc_lookup_failed" is increasing or not.

> show counter global name aho_alloc_lookup_failed

Name:           aho_alloc_lookup_failed
Value:          184328 <<<<<<---------------!!!!!!
Severity:       Warning
Category:       aho
Aspect:         resource
Description:    failed to alloc regex lookup

If you can see the above counter is increasing, please also check whether the "Regex Results" is depleted.

> debug dataplane pool statistics | match "Regex Results"
[12] Regex Results             (13272):     15/8192     0x8000000109219180


If your firewall matches to the above symptoms, the issue might be hitting to PAN-189468.
Please open a new support case via CSP with uploading tech-support file, if you need our assistance for checking the condition.


Resolution

  • This issue was fixed in PAN-OS 9.1.14 and 10.0.10-h1. Please upgrade PAN-OS if you are using affected versions (PAN-189468 only affects on 9.1.13 or 10.0.10).
  • To release "Regex Result" memory pool on affected PAN-OS, you have to reboot the firewall.



Hello in 9.1.14-h1 version this bug is fixed. After upgrade no issue seen.

Have a nice day.

UP

Thx for your reply,got it !

I'm not seeing anything in the 9.1.14-h1 release notes which would indicate this issue has been addressed.  May I ask what PAN-OS version you upgraded from?

L1 Bithead

Its PAN issue ID PAN-189468 and it was fixed in 9.1.14

As stated previously in this thread, 9.1.14 did not address the resource issue for us....hence my reason for asking.

show counter global name aho_alloc_lookup_failed always shows 0 even in 9.1.13 when the issue reported . & debug dataplane pool statistics | match "Regex Results" shows 8192/8192 on 9.1.13 -h3 , 9.1.14 h1 on 5220 . this indicate a similar unknown bug in 9.1.14 -h1 .

  • 1 accepted solution
  • 18894 Views
  • 17 replies
  • 2 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!