Overloading 5220 with 9.0.x

[Alex_Samad]

Hi

I update my firmware from 8.1.10 to 9.0.5

now I can bring my 5220 to its knees with my mailist run

So email consist of pdf attachment - approxy 3M.  but about 4K emails all around the same time

This wasn't a problem before on the 8.1.10 .. but on 9.0.5 cpu hits 100% and my latency through the box goes from <1ms to 2-3s+ which makes things crash 😞

I have put in a rule for my maillist server to no longer be content checked, but, I don't want to allow that for all email, I wouldn't mind ratelimiting it from the PA side of things, else somebody could crash my network by sending lots of email with large attachments to me !

Can I ratelimit 1 app or how can i get back to the same behaviour I had under 8.1.10

A

NOTE - sory original put in 5020 - fat finger mistake - 5220 

[AnalysisMan]

In your case, Packet Buffer Protection (PBP) should work, and it will protect your OSPF connections. I had many cases under high CPU spikes, and Zone Protection & DoS Protection didn't really help in my cases (probably, in your case as well.)

My engineering generates aggressive traffic sometimes, and it easily spikes up high CPU on the firewall. It's impossible to control or rate limit it because they use this protocol today, but later they may use other protocols or applications.

Even if your case is a bug, you can only delay the situation by upgrading the PAN-OS. The high CPU event could be happening later by other protocols or applications.

I'm happy with the PBP solution since I applied it. Because it protects the firewall and never reaches 100% CPU usage.

Here is the link for PBP.

https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/zone-protection-and-dos-protection/zone-de...

[Alex_Samad]

Hi

my only problem with all of these protections is are they are based upon number of connections or number of byte or flow rate.

Thinks that don't corrolate to threat detection.

I still think the best thing is to say threat protection can only use 80% of cpu ....

from what I read about PBP

"

When packet buffer consumption reaches the configured 

Activate

 percentage

"

works on the amount of traffic coming in - which might not relate to the amount of work threat protection has to do !

[TerjeLundbo]

Is it the dataplane CPU usage that is100%? We had to downgrade our 5220 firewall cluster from 9.0.4 to 8.1.11 a couple of months ago because the packet buffer filled up 100% with our normal traffic, something that was not a problem in 8.1.x.

[Alex_Samad]

Hi

Thats very interesting. So ... forgive me I might use the wrong words.  But I believe the CPU was at 100% across all the cpu - left no head room to process any packets for other things like OSPF heartbeats or BFD . etc etc.

I believe support said this was a 9.0.x thing.

So we had 4k emails ... some some text and a PDF . that would send out at the end of day.  not a problem with 8.1.x or 8.0.x 

But 9.0.x shat itself.  so I took emails of the threat protection path .. I think thats silly. but I have no way to mitigate the problem with out rate limiting down to almost 0..

[Alex_Samad]

@TerjeLundbo was that a recommendation from PA support to go back ???

Thinking thats a pretty big step to make that much difference 

[TerjeLundbo]

@Alex_Samad , this was a recommendation from our VAR/partner after consultations with TAC. 9.0.x was unusable for us.

[Alex_Samad]

@TerjeLundbo Oh ... could they id what caused the excessive CPU usage ... and its it planned on being fixed.

Trying to get this info from Support or my SE is very very painful - so I am eager to get as much info as I can 

[TerjeLundbo]

@Alex_Samad for us the issue was not CPU load but packet buffers filling up. Unfortunately I don't have a TAC case id or issue id for this.