Palo Alto PA-5220 - Data-plane traffic stops intermittently for 20-30 min

[BereketEstifanos]

We have a PA-5220 which seems traffic through data-plane stops intermittently for 20-30 min comes back up by itself.

The issue does not affect our management access as we are using the dedicated management interface which from what I understand it has its own resources separate from the data-plane. On the gateway, Intrazone communication does not seem to be affected as devices on the same zone are able to communicate to their  gateway which resides on the PA. Interzone communications is the one that is affected. We can see interzone traffic hit the gateway and matches with the rules we have set, but for 20-30 minutes approximately 6 times a day the traffic seems to get lost somehow. The logs don't show any odd alerts and the resources utilization for the gateway shows close to 0% on average. I upgraded the code version from 9.1.4 to 9.1.14-h4 incase it is some type of bug and failover the gateway to the secondary incase it is hardware issue, but still the issue persist. Has anyone experienced this issue ?

[akuzhuppilly · Accepted solution]

Hello @BereketEstifanos 

If the issue was caused by the FW DP, both intrazone and interzone communication should have been affected.

Based on your description, only interzone communication is affected, which is unusual. The issue persists even after the upgrade.

Have you thought about investigating other areas in your network that might be causing this problem?

[BereketEstifanos]

Thanks for the input @akuzhuppilly You are right, the issue was actually caused by ASA which have outside interface on the same broadcast domain as the transit interface for the Palo Alto. Proxy arp was enabled on the ASA outside interface which was causing to create ip address conflict with the PA interface. Since we are not NATing anything on the ASA disabling proxy arp on the ASA seems to fix the issue and the network have been stable for more than 12hrs now. The weird thing is neither the ASA or the PA generated " Duplicate ip address" message which would have been helpful.

[BPry]

@BereketEstifanos,

Any particular reason you went with 9.1.14-h4? 9.1.15 addressed a dataplane issue (PAN-189114) and 9.1.16 addressed a dataplane CPU issue that doesn't sound applicable (PAN-193763) but certainly could be contributing. PAN-OS 9.1.16 is the present preferred release within the 9.1 branch, I always hate to say "upgrade and see if it goes away" ... but there's two contributing dataplane issues in newer released that specifically address dataplane related issues.

I also think a 20-30 minute failure is a bit odd however. I've got plenty of clients running PA-5200 series devices that haven't seen that sort of failure resolution time in that late of a release. In conjunction with pursuing an upgrade, I'd take note of what @akuzhuppilly mentioned and also be testing things outside of your firewall. An actual dataplane issue should be affecting both intrazone and interzone traffic, not just interzone traffic. I'd be taking a good look at everything else as well to ensure that I'm actually trying to fix the proper device. Not saying it isn't the firewall, but it also sounds like it could potentially be a broader issue across the network.