Feature Request: ECMP Path Monitoring

[Aewald785]

We are currently using ECMP to load balance to our two ISPs. Which works great. However since there is no path monitoring(Unless you set static routes). If something happens upstream and your peer doesn't go down the PANs will happily keep sending data out that interface without batting an eye. Causing half your customers to be very unhappy. If path monitoring was available this would fix a lot of problems I think.  

Also I would be happy to hear any suggestions on other ways to do this. The idea is to load balance between two ISPs and if there's a problem upstream to force all traffic through the other interface until it comes back up. 

[BPry]

@Aewald785,

If you want to actually put in a feature request you'll need to reach out to your SE and have them officially add it for you. 

Your kind of expected to have two routes defined and just use the route path monitoring for this scenario to actually handle the failover. 

[TomYoung]

Hi @Aewald785 ,

If you are not using static routes, you probably are using BGP with the default route only, the ISP should remove the default route if they have problems upstream.  You bring up a good point.  How many ISPs don't remove the default if they have issues?  If that is the case, what is the advantage of BGP over static routes if the dynamic default route doesn't work or if you cannot receive the full Internet routing table?

I really wanted to come up with a way to use BGP conditional advertisement to solve your problem, but it will remove the advertisement and not the local route.  https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClEUCA0

Thanks,

Tom

[Aewald785]

@TomYoung 
Yep! That's exactly what's happening.

I've been trying to figure out a good way to do this. I've seen plenty of posts on load balancing with PBF rules, but that's not true load balancing and not very dynamic. I like that thought process with the conditional advertisement though, it was a good idea! 

[TomYoung]

Hi @Aewald785 ,

If your ISP does not reliably remove the default route when they have issues, you can remove BGP and use static routes with path monitoring.  I wouldn't use PBF.  Static routes are more straightforward.

Thanks,

Tom

[EddieHerrera]

I am trying to design my network in a similar fashion. I was wondering why you don't have default static routes for your ISPs. You mentioned ECMP, but to my knowledge ECMP is a just a routing protocol that uses an algorithm to load balance the traffic to a destination with multiple paths. 

That being said, what routing protocol are you using to build those paths in your routing table? 

In our case we are using static default routes for outbound traffic through ISPs with path monitoring to at least 3 destination IPs. Now for our IPSec traffic I am stuck between static routes and ospf. I like that static routes have path monitoring, allowing you to ensure that routes are only added back when they meet your monitoring requirements. On the other hand I like that OSPF provides easier management overhead. We are a small network, but I still hate the idea of having to add a static route to each VR. 

I plan to test OSPF with ECMP to make sure that OSPF removes routes when they are not avaialbe and ECMP does not attempt to route traffice over those routes.

[ChrisMyers]

If path monitoring breaks with ECMP, you may have Strict IP Address Check enabled in your Internet Zone Protection Profile.  Path monitoring automatically uses the next-hop address of the route that you're monitoring (bypassing the route table), BUT when the path goes down then the route is removed from the routing table.  The Strict IP Address Check verifies that the source IP address (of the _reply_ to the path monitor packet) is routable over the _exact_ ingress interface.  PAN-OS then drops the reply packet because there is no route to the host you're monitoring via the ingress interface.

If you have static routes to your path monitor destinations, then the Strict IP Address Check succeeds.  Of course you lose access to the path monitor destinations when the route goes away, because you have static routes pointing at a non-working path.

The correct fix is to disable Strict IP Address Check in your Internet-facing zone(s).  Only the Internet, though.

[KBukovan]

I have this setup using path monitoring with just the next hop, my ISP GW as the host to monitor. I assume if they are up, their network is good. Using other IPs like 8.8.8.8 etc...always one line would end up going down for no apparent reason. I get on the firewall ssh, ping from the interface to all of the IPs in the list and they work fine. For whatever reason, ECMP and path monitoring are bugged. Would rather not use path monitoring yet still have it load balance and know when a line it down to route all traffic through the other ISP. Guess I was spoiled with my Sophos UTM where it handled this automatically by checking a box and setting up a preferred line/prioritized one and you can set monitoring. Never had a single issue with proper load balancing and wackiness like Palo products. Frustrating for such basic stuff to work.