Overloading 5220 with 9.0.x

cancel
Showing results for 
Search instead for 
Did you mean: 

Overloading 5220 with 9.0.x

L4 Transporter

Hi

 

I update my firmware from 8.1.10 to 9.0.5

 

now I can bring my 5220 to its knees with my mailist run

So email consist of pdf attachment - approxy 3M.  but about 4K emails all around the same time

 

This wasn't a problem before on the 8.1.10 .. but on 9.0.5 cpu hits 100% and my latency through the box goes from <1ms to 2-3s+ which makes things crash 😞

 

I have put in a rule for my maillist server to no longer be content checked, but, I don't want to allow that for all email, I wouldn't mind ratelimiting it from the PA side of things, else somebody could crash my network by sending lots of email with large attachments to me !

 

Can I ratelimit 1 app or how can i get back to the same behaviour I had under 8.1.10

 

A

 

 

NOTE - sory original put in 5020 - fat finger mistake - 5220 

22 REPLIES 22

May I ask how you managed to update a PA 5020 to PAN-OS 9.0?

We were told PAN-OS 9.0 is not supported on PA 5000 series.

apologies 5220

@Alex_Samad don't get me wrong because of this question but as this somehow seems like a bug of 9.0, what about a downgrade to 8.1? Or did you upgrade to 9.0 because of a new feature that you need?

I agree with you that the firewall should not reach 100% cpu usage because of such a connection, but I mean also a 5220 has it's limit an when this is reached - I know this is not the situation you have - you have to enable some protections - as described by @BPry - on the surounding systems.

 

And the this of course this does not help you either, but do nut upgrade to a new major version prior to x.y.7 (or better x.y.8)

Hi

 

I am not sure if its a bug.

 

Why the move to 9.0  . thats the PA recommendation

Why 9.0.5 Thats the recomemnded release 

 

I would hope that PA wouldn't recommend things that don't work !

 

The protection should be, in my opinion, that you can apply caps to CPU limits for type or process on the PA.

 

Threat prevention shouldn't take up 100% over and above packet routing .. which is what happened.

 

Yes you can try and limit traffic / data to / through the PA but that is fraught with danger - again don't fix the symptoms fix the issue.

 

 

Any way I am happy - sort of - with my change - I have remove content checking ... I would like it on but I can live with out it for now

 

 

 

L4 Transporter

So ran into this problem again . but on my PA850 - much smaller PA.

 

So users connecting to file server.

 

Normally the PA's id's traffic as ms-ds-smbv3 no problem it doesn't get checked so no issue.

But the last one that went through was ID'ed as rss. not sure how that worked and it ifs under my catch all.

 

It was 27G session

 

so it tanked my CPU - stoped OSPF deamon replying to hello's. 

 

basically killed the Firewall and brought down the network in the office

 

To me this is a Bug - rather big one ... not being able to limit the amount of cpu time that is allocated to threat protection

 

 

In your case, Packet Buffer Protection (PBP) should work, and it will protect your OSPF connections. I had many cases under high CPU spikes, and Zone Protection & DoS Protection didn't really help in my cases (probably, in your case as well.)

 

My engineering generates aggressive traffic sometimes, and it easily spikes up high CPU on the firewall. It's impossible to control or rate limit it because they use this protocol today, but later they may use other protocols or applications.

Even if your case is a bug, you can only delay the situation by upgrading the PAN-OS. The high CPU event could be happening later by other protocols or applications.

I'm happy with the PBP solution since I applied it. Because it protects the firewall and never reaches 100% CPU usage.

 

Here is the link for PBP.

https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/zone-protection-and-dos-protection/zone-de...

 

--
"The Simplicity is the ultimate sophistication." - Leonardo da Vinci.

Hi

 

my only problem with all of these protections is are they are based upon number of connections or number of byte or flow rate.

 

Thinks that don't corrolate to threat detection.

 

I still think the best thing is to say threat protection can only use 80% of cpu ....

 

from what I read about PBP

"

When packet buffer consumption reaches the configured 

Activate

 percentage

"

 

works on the amount of traffic coming in - which might not relate to the amount of work threat protection has to do !

 

Is it the dataplane CPU usage that is100%? We had to downgrade our 5220 firewall cluster from 9.0.4 to 8.1.11 a couple of months ago because the packet buffer filled up 100% with our normal traffic, something that was not a problem in 8.1.x.

Hi

 

Thats very interesting. So ... forgive me I might use the wrong words.  But I believe the CPU was at 100% across all the cpu - left no head room to process any packets for other things like OSPF heartbeats or BFD . etc etc.

 

I believe support said this was a 9.0.x thing.

 

So we had 4k emails ... some some text and a PDF . that would send out at the end of day.  not a problem with 8.1.x or 8.0.x 

 

But 9.0.x shat itself.  so I took emails of the threat protection path .. I think thats silly. but I have no way to mitigate the problem with out rate limiting down to almost 0..

 

 

@TerjeLundbo was that a recommendation from PA support to go back ???

 

Thinking thats a pretty big step to make that much difference 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!