Packet buffer protection - PA5220 vs PA5410

[michelealbrigo]

I've recently upgraded my firewall from a PA-5220 pair to a PA-5410 pair. The firewalls were on the same PanOS version (10.2.4-h2) and with the same configuration. This was the original configuration for PBP at the upgrade time:
The 5220 wasn't logging any PBP intervention, as you can see here (there's some sporadic intervention by zone protection profiles, but I consider it as normal):

When we switched to PA-5410s (same OS and config), the firewall started logging PBP protection events:

I do not expect this to be due to a change of traffic nature, since there's roughly one minute between the last event logged on the 5220 and the first one on the 5410. Also, with some log analysis, the events fall in the time interval between the firewall change and when I deactivated PBP:
I turned PBP to monitor, and set it to capacity-based, and the alerts went away:

Has anyone got an explanation for this? Why is there such a difference between 5220 and 5410 in latency-based PBP?

On a side note: by being focused on routing issues and other migration-related stuff, and "one-man-band" on the issue, I completely lost the meaning of the threat name from the Dashboard widget, so it took me a bit longer-than-optimal time to realize that PBP was the source of many problems experienced by my users during those days. Could this kind of intervention be referenced in the "session end reason" field of the logs, instead of "aged-out"? I only figured that out because an "aged out" session I was investingating showed a very large (and strange) difference between received and sent data:

[BPry]

@michelealbrigo,

When you switched out the hardware did you clear out the existing ARP entries for the connected switch(es)/router(s)? Latency being introduced on a new hardware install I'm always going to lean that way out the bat. I'll also just note that with the PA-5410 your on a completely different platform, so you could still be running into a bug that simply wasn't present on your PA-5220. Not saying you're running into a bug, but in the event you haven't engaged TAC because you think it isn't a bug it would be a good idea to engage them. While I'm unaware of a bug matching this on the 5400 series, only employees have access to Jira to check all bug reports. 

[michelealbrigo]

I did not clear the arp cache on the routers, but the firewall only talks with a small number of L3 switches (3 physical switches with some VRFs, make it less than 20 in total), and all of them only have the firewall on the uplink VLANs. I.e. the links are an improper "point-to-point", all /24s with just the switch uplink IP and the firewall downlink IP. Also, this situation went on for days before I realized that, so that should be beyond the ARP cache duration.

As for the TAC request: I'm still on an "explorative" phase, the 5410 deployment isn't complete yet, so I wouldn't be able to be consistent over the days on any check I might be required to perform. I'll definitely give PBP a go when everything is in its place, and open a proper ticket if I can consistently replicate the problem at will.

[ocardiel]

Hi Michele,

I have just the same issue... were you able to solve or find an explanation for this issue?

thank you in advance for your help.

[michelealbrigo]

No, sorry: no solution, neither explanation. In the meantime, I've moved up some releases (10.2.6 currently), but never come back to check if latency-based PBP is back working as intended. A call with the TAC only provided some suggestions on tweaking the latency activation values, but PBP was still acting wrong, so I basically gave up on that.

Our PBP is configured as capacity-based, at the moment (50% alert, 80% activate), apparently without issues: 5410s are kinda oversized for our deployment, when running under normal load, and that's also what made those latency-based PBP triggers strange.

[ocardiel]

Thank you very much for your answer, Michele. I appreciate it a lot!