Having to reset the dataplane frequently

[grkchr]

Hello,

We've been having an issue in our environment where we need to reset the dataplane because randomly packets will traverse our rules and start getting denied. We aren't sure why this is happening or what's causing it. What I'd like to know is if anyone could shed some light on how we can go about troubleshooting.

Let me know what info you may need.

Thanks,

C

[kprakash]

Good Morning,

I am afraid if its an issue with the dataplane, but rather the way the traffic doesnt match the polices configured on the box. So does the traffic match the intended rule sometimes and matches the deny rule the other times? If not, you can determine what application the traffic is matching and write a specific rule for it. You can create an "any, any" permit rule and place it above your clean up rule, and then look for the traffic logs for those sessions. The traffic logs will specify the from and the to zones, the source and the destination IPs, the usernames, the matched application, etc. You can then use this information to create a more specific rule,and place it appropriately on the security rule list.

Also when you mean reseting the dataplane, are you rebooting the device or just issuing a command to restart the dataplane?

BR,

Karthik

[grkchr]

Morning Karthik,

I also thought it was an issue with our rules, but the rules work 99% of the time and resetting the dataplane fixes the issue whenever this occurs. The reason we notice is because all of our inbound traffic starts getting denied. All the traffic logs look normal. Is there any deeper digging that can be done...I'm thinking we may need to do a packet capture the next time this happens.

We are going to Device-->Restart Dataplane through the web gui.

C

[kprakash]

Is it a proprietary application that is being denied ( traffic for which we do not have signature for ) ? If that is the case, we can create a custom app for the traffic in question, and apply it under an app override policy and the security rule that it is to match.

Are you facing this issue for any traffic, and how frequent is it?  We also want to check that the dataplane is not overwhelmed. You can issue the command

>show running resource-monitor

>show session info

to verify that the data-plane is healthy. The first command gives the sanpshot of the dataplane for a specific duration. The second command gives the number of active sessions and the throughput.

Alternatively you can also monitor the ACC to look at which app is eating up a lot of sessions and bytes.

BR,

Karthik

[kprakash]

That definitely sheds some more information. So if its inbound traffic, we are doing a destination NAT to a server, arent we?

can you give us the output of the following commands:

>show running nat-rule-ippool show-freelist yes show cache yes rule <destination-nat-rule>

BR,

karthik RP

[kprakash]

Also when it stops working, we better enable filters and packet capturing and look at the counters, the pcaps and the debug logging to understand the issue. I would recommend opening a ticket with us to investigate the reasons behind the drops.

BR,

Karthik