Conditional Advertisement / BGP Failover with Dual ISP — How to Remove ISP1 Routes on Internet Loss?

[Austin_Mascarenhas]

Hi all,

I’m running a dual-ISP setup on a PA  with BGP to ISP1 and ISP2. My goal is:

1. Monitor ISP1 default route / Internet reachability.

2. If ISP1 becomes unusable, I want all traffic to fail over to ISP2.

3. I am advertising an IP pool to both ISP1 and ISP2 for incoming traffic, with AS-path prepending applied to ISP2 so that incoming traffic prefers ISP1. Ideally, I would like all ISP1 routes to be withdrawn when the upstream Internet fails.

Here’s what I’ve tried and observed:

• Conditional Advertisement:
  I configured a policy on ISP2 to advertise the IP pool only when ISP1 default is missing. Works in principle, but I cannot cancel advertisement to ISP1 just because ISP1 stops sending a default route.

• Path Monitoring (pinging a remote IP):
  This removes the route for outbound traffic, so outgoing connections failover to ISP2. However, the IP pool advertisement is still sent to ISP1, so incoming traffic continues to fail.

Questions:

1. Is there any way in PAN-OS to completely withdraw all BGP routes and bring down ISP1 session when the Internet behind ISP1 fails but the peer IP is still reachable?

2. Would combining conditional advertisement / AS-path prepending achieve practical failover for both incoming and outgoing traffic?

3. Are there any recommended workarounds in PA for this scenario that don’t involve extra hardware or ISP cooperation?

Appreciate any guidance or shared experiences.

[TomYoung]

Hi @Austin_Mascarenhas ,

I see that no one has replied to your question in a few days; so, I will give it a shot.

Is there any way in PAN-OS to completely withdraw all BGP routes and bring down ISP1 session when the Internet behind ISP1 fails but the peer IP is still reachable?

Not that I know. This is where the full BGP routing table comes into play, but the PA-Series cannot handle that many routes.  Purchasing a couple of BGP routers may simplify the process.

With regard to oubound traffic, I doubt you can rely on your ISP to withdraw the default route if they have Internet connection problems.  In most cases, the route is not withdrawn and the traffic gets blackholed.  If all you are receiving is the default route, you could tell the ISP not to send it and use path monitoring with a static default route. 

Would combining conditional advertisement / AS-path prepending achieve practical failover for both incoming and outgoing traffic?

I have never seen AS path prepending achieve 100/0 load balancing for incoming traffic even with the max prepends.  I don't see any advantages to combining it with conditional advertisement.

Outgoing traffic would be load balanced by BGP weight or local preference.  https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClszCAC

Are there any recommended workarounds in PA for this scenario that don’t involve extra hardware or ISP cooperation?

I think path monitoring with default static routes would be more reliable than BGP for outgoing traffic.  For incoming traffic with BGP you would have to assume the ISP would withdraw the default route if they had internet issues.  With regard to conditional advertisement, how can 0/0 be the withdraw prefix if it is received from both ISPs?  I think the best way is to use ECMP and allow the outbound and inbound traffic to load balance.  You definitely want to check the Symmetric Return box under ECMP so that return traffic for incoming connections is sent out the same interface.

Thanks,

Tom