Cascading PBF policies for traffic fail over between two sites both using PAN firewalls?

[AaronMarshall]

My goal is to sent traffic to/from two sites with three levels of redundancy.  Both sites have redundant PA-3020s.  First, traffic goes through a L2 point-to-point circuit.  If this fails and is disabled, an IPSec tunnel takes over.  Lastly, a third IPSec tunnel takes over should the previous one fails.  All interfaces and tunnels will be in the same security zone.  

Here is what I'm thinking for set up:

Primary L2 point-to-point circuit

- A unique virtual router for the L2 circuit interface is set up on both PANs including static route for the other site's subnets.

- A PBF is set up on both sides to send traffic to the other site.  Monitor is set to wait-recover and set to disable if the monitor IP is unavailable.  Enforce Symmetric Return is active.

- If the PBF is disabled due to a monitor failure, another PBF in the sequence will route traffic through an IPSec tunnel.

- Through wait-recover, if the monitor heartbeat is active traffic fails back to this PBF since the rule is reenabled.

Secondary - IPSec tunnel

- A unique virtual router is set up for the tunnel interface similar to the previous set up.

- A PBF is set up on both sides sending the traffic through the tunnel exactly like the L2 PBF.

- If this fails over due to another PBF montior failure, traffic will route through the tertiary IPSec tunnel.

- Through wait-recover, if the monitor heartbeat is active the PBF rule will no longer be disabled.  I assume that traffic will route through this tunnel unless the original L2 PBF is renabled through the wait-recover. 

Tertiary - IPSec tunnel
IPSec tunnels on both sides reside within the primary default Virtual Router. Static routes exist with that virtual router to route traffic through the tertiary tunnel.

Questions:

With complementary PBFs on both sides, will the PBF "Enforce Symmetric Return" feature ensure that I won't face an issue where traffic on one PAN and interface will try to be returned from the other unit on a different interface or tunnel?

Since PBFs operate from top to bottom precidence, will the secondary rule sit idle until the L2 rule above it is disabled?

Will the wait-recover delays (currently set at the default Interval 3 and HB 5) cause inconsistent behavior on both sides which and lead to  potential symmetric return issues?

Should the tertiary IPSec tunnel also reside in a PBF instead of being in the default virtual router? 

Thank you!
Aaron

[pulukas]

With complementary PBFs on both sides, will the PBF "Enforce Symmetric Return" feature ensure that I won't face an issue where traffic on one PAN and interface will try to be returned from the other unit on a different interface or tunnel?

This should work as you outline.  And if it does not, this won't be critical because you have placed the three paths into the same zone on the firewall.  So even if the traffic is asymmetrical the session will still match and the traffic will be accepted.

Since PBFs operate from top to bottom precidence, will the secondary rule sit idle until the L2 rule above it is disabled?

This will work, I would prefer to call it "not hit" as opposed to "idle" but that is just symantics.  The idea is the same the rule order is what is dictating the path.

Will the wait-recover delays (currently set at the default Interval 3 and HB 5) cause inconsistent behavior on both sides which and lead to  potential symmetric return issues?

Very likely, the chances of the recoveries being identical is small.  But as noted above your zone setup protects the traffic in any case even when not symmetrical.

Should the tertiary IPSec tunnel also reside in a PBF instead of being in the default virtual router? 

Either would work but putting this in the default VR is simpler and I tend to like simplier solutions.

Thoughts:

I wonder if you had considered setting this up with straight up normal route preferences instead of using PBF.  You seem to be using static routes, so if the static routes just had the preferences in order of your paths and you setup the path monitoring to bring down the interfaces when the path is not valid, you would get the same failover sequence and behavior.  And the configuration would be simplier still.