Multiple ISP with PBF and BGP

[khsieh]

I am trying to setup services on a secondary ISP link and make those services available even if the link fails via another path to another data center that eventually ties to that same ISP.

Here are the details. In data center B, I have three public ISP links. The first two links are to ISP 1, and the second link is to ISP 2. All of these public IP subnets are less than /24, so I cannot publish anything using my own public ASN.

I want to move some services onto ISP 2. If the link fails, I want the traffic to egress and ingress through an IPSec tunnel that I have to data center A. Data center A has a connection to ISP 2, so I can advertise my subnet out that link with some ASN padding. That works fine.

Because I need to route just some devices out the second ISP, I am using PBF to do that, and therein lies my challenge. I am having problems covering my bases to detect problems with my link to ISP 2. I have setup my PBF rule to ISP 2 to monitor an IP address. I have tried to monitor both the next hop, and an IP address out on the Internet using a static route. I am trying to duplicate reliable static routing like I would with Cisco. The problem is that my PA firewall is doing the monitoring with the IP address that is on the same subnet as the next hop. This is confirmed through packet capture, and is true even if the first IP address of the interface is on different subnet.

My problem with the monitoring is that it detects the link being hard down, but I haven't been able to get it to detect a problem with BGP. Hey, BGP problems happen, and I would like to not send my traffic down an asymmetric black hole if I can help it.

The reason why the link monitoring fails to do what I want is because it uses the transit network for the source IP. Even without BGP active, that IP address will always be in the routing table of ISP 2 and hence I will get the return traffic is the link has basic connectivity. If I could use my BGP subnet as the source IP, then return traffic would only arrive though ISP 2 if my BGP connection was working, and the monitor would do what I need it to do.

I have also tried to PBF the egress traffic to a different virtual router, and then just let the routing table of the second virtual router do it's thing, but I was unable to get that to work. I think I was running into problems just trying to PBF traffic to another virtual router. I had no issues setting PBF to send traffic out a specific interface on the other virtual router, but I couldn't just drop it off to the other virtual router and then let dynamic routing do its thing.

I am looking for network level solutions without resorting to things like updating public DNS and doing NAT to different IPs depending on which ISP circuit was available. Even still, I wouldn't be able to detect when BGP was down if the link was up at L3, so it isnt even useful.

Thanks for reading.

-Kevin