Inbound Policy-Based Forwarding Issue - Intermittent loss of connectivity

[GrantCampbell4]

Hello,

Got a strange one, that I am hoping someone with deep knowledge of PBF and symmetric return can advise on.

We have two (2) virtual-routers due to two different ISPs.  The history of it is we are migrating off of one ISP to finally decommission it.

Most of the internal DMZs are on VR1

New VR2 is the new ISP

We have eBGP between VRs using loopbacks to share hundreds of internal routes

We have multiple outbound PBFs rules pushing traffic out form VR1 to VR2, which are zone type PBF rules

We have several inbound PBF rules, which are all interface type PBF rules pushing traffic to VR1 from VR2 all with Enforce Symmetric Return

One of these inbound services and PBF rules is intermittently failing.

Packet captures show :

Client to server reaching the Palo Alto in VR2

Palo Alto forwards that onto destination server in VR1 and applies the required NAT

Server returns TCP-SYN-ACK data to Palo Alto

However Palo Alto does not send that TCP-SYN-ACKs to client and is seen in the drop captures

At this stage multiple TCP retransmissions ensure from both the client and server but are also dropped until a TCP RST is sent from the server, which again is dropped

Session browser shows session is built and NAT and symmetric return flag checked

This is an intermittent issues last several hours.  Other inbound PBF rules and services do not seem to be affected.

PA support are currently baffled and have not yet stumped up any real next steps.

We moved from PA-3020s on 9.1.x back in Nov 2023 to new PA-5410s on 10.2.5 when this issue began.

Are there limits as to the number of PBF rules

Are there limits to the number of outbound PBF rules and inbound rules 

CPU showing as low at 5-10% on the data plane and session table is 39500 out of 5000000 at it's peak

Health check on physical network through FW, switches and ISP infrastructure shows as all clear, with no errors. (We would see errors in other services that share this infrastructure)

Looking for any advise and advance cli commands that could help me troubleshoot this problem.

Regards

[iarobertson]

Hello @GrantCampbell4 - I am going to suggest raising a TAC case for this one; if you've seen the behaviour introduced between 9.1 and 10.2, and no other changes were made, a TAC case is the appropriate way forward.