topic Re: session end reason resources-unavailable for all traffic in General Topics

session end reason resources-unavailable for all traffic

miaocongcong — Mon, 28 Mar 2022 03:30:43 GMT

Hi,Guys

The customer's network recently experienced an outage, and found all the session end reason was resources-unavailable ;

I exec the comand " debug dataplane pool statistics" and found there is a parameter in the software pool called Regex Results that has been exhausted. Compared with normal firewalls, this value should not be reduced. I also tried to execute “set deviceconfig setting session resource-limit-behavior bypass” , but this solution is not acceptable to customers.
At present, the customer's system version is 9.1.13 of the PA3000 series, which has been rolled back to 9.1.12, and the parameters are normal
Does anyone know if this is a bug?

Customer needs an official explanation，thanks in advance！

Re: session end reason resources-unavailable for all traffic

PavelK — Mon, 28 Mar 2022 05:26:10 GMT

Thank you for the post @miaocongcong

I experienced exactly the same issue in mid of February after I upgraded PA-3220 to 9.1.13. The command you executed: "debug dataplane pool statistics" is the right way to drill down the issue. As an immediate measure we failed over the Firewall and rebooted the one that was having an issue and opened a TAC case. After spending some time on the call, TAC advised to downgrade the Firewall that solved the issue, however we could not get anything official regarding bug. Based on the input from TAC, in version 9.1.13 there were some issues that engineering team is looking into, but there is no official bug id for this.

Kind Regards

Pavel

Re: session end reason resources-unavailable for all traffic

upelister — Thu, 05 May 2022 10:56:34 GMT

Hello with 9.1.13-h3 we hit this bug, we are waiting TAC response. Downgrade means SSL loop vulnerability.

https://security.paloaltonetworks.com/CVE-2022-0778

Re: session end reason resources-unavailable for all traffic

upelister — Thu, 05 May 2022 14:43:47 GMT

Do you have any idea why "set deviceconfig setting session resource-limit-behavior bypass" solution is not acceptable to customers?

Re: session end reason resources-unavailable for all traffic

ValleyComm — Wed, 18 May 2022 16:40:11 GMT

Because when you bypass, you are bypassing L7 threat protection.

Re: session end reason resources-unavailable for all traffic

PavelK — Thu, 19 May 2022 00:13:12 GMT

Today, PAN-OS 9.1.14 was released. This issue was addressed in this release: PAN-189468

https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-release-notes/pan-os-9-1-addressed-issues/pan-os-9-1-14-addressed-issues#panos-addressed-issues-9.1.14

Kind Regards

Pavel

Re: session end reason resources-unavailable for all traffic

bvandi — Mon, 23 May 2022 16:11:02 GMT

9.1.14 does not appear to have successfully addressed this issue. We upgraded our 5220s over the weekend and are now facing the same issue.

admin@Firewall(active)> debug dataplane pool statistics | match Regex

[12] Regex Results (13272): 1020/8192 0x800000042bfda380 205

Re: session end reason resources-unavailable for all traffic

PavelK — Tue, 24 May 2022 06:35:00 GMT

Thank you for feedback and sorry to hear that @bvandi

to be honest, there is not much I can add to this. Myself, I am no longer facing this issue and can't comment on this anymore. At this point I would open a ticket to TAC to address this issue.

Kind Regards

Pavel

Re: session end reason resources-unavailable for all traffic

William-Wu — Tue, 24 May 2022 17:49:48 GMT

From the Article: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000004ODICA2&lang=en_US%E2%80%A9

Symptom
After upgrading PAN-OS to 9.1.13 or 10.0.10, unexpected traffic failure may occurs and traffic log shows the session end reason "resources-unavailable".

Environment

All platforms including VM firewalls
Firewalls running on PAN-OS 9.1.13 or 10.0.10 (not affected to other PAN-OS versions)

Cause
If you can see the session end reason "resources-unavailable" under traffic log without resource usage spike after upgrading PAN-OS to affected versions, please check whether the counter "aho_alloc_lookup_failed" is increasing or not.

> show counter global name aho_alloc_lookup_failed

Name:           aho_alloc_lookup_failed
Value:          184328 <<<<<<---------------!!!!!!
Severity:       Warning
Category:       aho
Aspect:         resource
Description:    failed to alloc regex lookup

If you can see the above counter is increasing, please also check whether the "Regex Results" is depleted.

> debug dataplane pool statistics | match "Regex Results"
[12] Regex Results             (13272):     15/8192     0x8000000109219180

If your firewall matches to the above symptoms, the issue might be hitting to PAN-189468.
Please open a new support case via CSP with uploading tech-support file, if you need our assistance for checking the condition.

Resolution

This issue was fixed in PAN-OS 9.1.14 and 10.0.10-h1. Please upgrade PAN-OS if you are using affected versions (PAN-189468 only affects on 9.1.13 or 10.0.10).
To release "Regex Result" memory pool on affected PAN-OS, you have to reboot the firewall.

Re: session end reason resources-unavailable for all traffic

upelister — Thu, 07 Jul 2022 19:56:55 GMT

Hello in 9.1.14-h1 version this bug is fixed. After upgrade no issue seen.

Have a nice day.

Re: session end reason resources-unavailable for all traffic

miaocongcong — Sun, 10 Jul 2022 19:03:30 GMT

Thx for your reply，got it ！

Re: session end reason resources-unavailable for all traffic

bvandi — Fri, 15 Jul 2022 15:28:23 GMT

I'm not seeing anything in the 9.1.14-h1 release notes which would indicate this issue has been addressed. May I ask what PAN-OS version you upgraded from?

Re: session end reason resources-unavailable for all traffic

ValleyComm — Fri, 15 Jul 2022 15:31:23 GMT

Its PAN issue ID PAN-189468 and it was fixed in 9.1.14

Re: session end reason resources-unavailable for all traffic

bvandi — Fri, 15 Jul 2022 15:34:13 GMT

As stated previously in this thread, 9.1.14 did not address the resource issue for us....hence my reason for asking.

Re: session end reason resources-unavailable for all traffic

agil.puthukudynellikka — Wed, 20 Jul 2022 09:12:57 GMT

show counter global name aho_alloc_lookup_failed always shows 0 even in 9.1.13 when the issue reported . & debug dataplane pool statistics | match "Regex Results" shows 8192/8192 on 9.1.13 -h3 , 9.1.14 h1 on 5220 . this indicate a similar unknown bug in 9.1.14 -h1 .

Re: session end reason resources-unavailable for all traffic

IljaCigna — Thu, 10 Nov 2022 03:35:14 GMT

on 9.1.14-h4 it is still happening (VM 300) and started when we were on 9.1.3 with the key difference being that the counter mentioned in either version has not been increasing, remained at 0 at all times including during sessions being dropped. In our case this happens at very specific times of the day (around 8 AM and 8 PM every day). It is for very specific traffic between two hosts which however for the rest of the day has no issues. Something clearly triggers this within PAN OS and it goes away when that specific traffic is no longer passing through. We could recreate the issue at will. It happens to sessions of any total size, just a few kilobytes as well as tens of megabytes long. We have logged this with TAC a couple of days ago.

Re: session end reason resources-unavailable for all traffic

AaronAxvig — Thu, 16 Feb 2023 17:23:38 GMT

I am still seeing this on 9.1.14-h4 as well. FTP transfer of 4.5MB file jams up after 2.5MB with resources-unavailable logged.

Re: session end reason resources-unavailable for all traffic

RomainSalmon — Tue, 23 May 2023 09:48:27 GMT

Hello,

We had this issue for our 5250 despite of the upgrade. After opening a TAC case, the session ends with a resources-unavailable reason as the ctd-pkt-proc-loop counter is being triggered : (show session info)
tracker stage l7proc : ctd pkt loop
end-reason : resources-unavailable

Cause of the issue:- During the packet processing, ctd reaches the max number of loops which triggers the ctd_pkt_proc_loop counter and ends the session with resources-unavailable.

Resolution:- Increasing the number of loops would allow ctd to finish processing the packet buffers so the session will not end due to reaching the max number of loops.

- Please change the number of allowed loops to 8190 using the following commands.
set system setting ctd pkt-proc-loop-low 8190
set system setting ctd pkt-proc-loop-high 8190

- After changing loop values revert resource limit behavior to drop followed by a commit.
>configure
# set deviceconfig setting session resource-limit-behavior drop
# commit
- After commit is complete, verify new loop values using the following command.
>show system setting ctd state | match loop

I hope that gonna help some people !