topic Re: High availability failover due to high dataplane usage in General Topics

High availability failover due to high dataplane usage

mmbengue — Thu, 05 Oct 2017 06:46:44 GMT

our internet went down a few weeks ago when our primary PA failover to a secondary PA. We found out, after doing some research and investigative work, that this was due to the amount of new session created, which cause the PA to use the slowpath and access more cpu resource. Once we failedover, we had internet access for about 5-10 minutes and then suddently we lost internet access. After talking to tech support, we came to a conclusion that this might have been to due arp. We have about 25 static NATs, and 3 DNATs, could this have been the cause? if so why did we have internet for a while and suddently lost connection?

our failover condition is based on link monitoring, trust, untrust, and path monitoring, which is our gateway.

Why didnt we fail back to the active if we lost access to the internet- our path was never down on our secondary FW.

We're running 8.0.2

Re: High availability failover due to high dataplane usage

reaper — Thu, 05 Oct 2017 07:48:23 GMT

Sounds like your device may have been flooded. this would explain your connectivity came back for a few minutes while the newly active secondary firewall's session table/resources were rapidly depleting.

This could be an outside attack or an inside burst (example: simultaneous triggering of windows update on a lot of internal devices)

one way to prevent this type of issue from taking down your firewall is to enable Zone Protection profiels that will start discarding packets at a certain packet rate or will implement syn cookies to prevent malicious flooding of tcp sockets

did you happen to collect a techsupport file right after the incident on your primary? if so you could go take a look at the dataplane resources in the dataplane logs to see if your packet descriptors were filling up or software pools draining

is there any additional information you can share?

you mentioned ARP, could you elaborate on this conclusion?

Re: High availability failover due to high dataplane usage

VinceM — Thu, 05 Oct 2017 12:24:31 GMT

Hi,

Huge amount of new session ... normally, session are sync between cluster member then ...

Which model of palo are you using ??

According my experince, path monitoring is not .... always efficient.

Maybe when your first palo carshed, failover happen but path monitoring was not up on backup .. then no internet ....

Hope help.