2xISPs and 2 VPN tunnels - tunnel failover issue

[Andreikin]

Hello Team,

I am running into an issue with our setup. We have single PA460 box connected to 2 ISPs same time, i.e. Ethernet1/3 is to ISP1 and Ethernet1/4 to ISP2. We are running 2 default routes setup like this with ECMP enabled, so traffic is been load-balanced betwen 2

ISPs with a hash based on source. We also have 'strict source path' option enabled in ECMP settings. Both default routes have monitor enabled, so if for example Google DNS IP is not reachable over this ISP, it will be removed from routing table. On top of that we are running 2 VPN tunnels from this box, each of those using one of the physical interface IP address bellow as source. Both VPN tunnels are been terminated on the same remote Public IP. 

When 2 ISPs are running normally and traffic is going through those, with 'strict source path' option we have VPN tunnel traffic bound to the same VPN tunnels they are sourced from. So encrypted traffic for VPN1 is always going out through Etherent1/3 and encrypted traffic for VPN2 is going through Ethernet1/4. 

But, for example, if ISP1 is DOWN, we see that default route through ISP1 via Ethernet1/3 is down and removed from routing table, which is expected. What we did not expect is we see in traffic captures that Palo appliance is trying to bring up VPN1 through Ethernet1/4, while still using public IP of Ethernet1/3. That VPN is not obviously coming up as public IP of Ethernet1/3 is not reachable externally any longer, but an actual behavior is concerning and in few corner cases might lead to a weird case that VPN1 will be established using Ethernet1/4 as egress interface. I've tried to use VPN monitoring profile with 'wait-recover' option, but it seems to lead to the same issue. 

Is there any hints or pointers you can give me in this situation and how to avoid it?

Thanks!

[OtakarKlier]

Hello,

Do the VPN tunnels terminate at the same endpoint device, on the other end?

Regards,

[Andreikin]

Hi @OtakarKlier from our side and from remote side they do terminate on a very same device. We just have two tunnels going over 2 ISPs for a redundancy purposes. 

[OtakarKlier]

Hello,

I'm gonna guess that this is because of the default routes and IPSec tunnel config. The tunnel is trying to use the IP of the down interface via the up interface. Cant think of a way to prevent the IPSec tunnel with IP of the down interface to not use the up interface etc. I dont run ECMP so cant say I've run into this. I usually just use OSPF routing for the tunnels and for the ISP use Policy Based Forwarding to prefer one over the other, so just failover.

Hopefully someone else can jump in and share their thoughts. @BPry @reaper @aleksandar.astardzhiev @TomYoung 

[Andreikin]

@OtakarKlier yes, that happens indeed because of the fact, that 'strict source path' thingie only works when you have ECMP in effect, which is when you have at least 2 routes to the same destination and same metric. When one of the ISPs is down, than 'strict source path' is no longer in effect I guess.

But to your answer, can you elaborate what are you using PBF for? AFAIK you can only force traffic THROUGH firewall to be affected by PBF and not traffic source/destined to it. 

[OtakarKlier]

Hello,

Policy based forwarding take effect prior to the virtual router. So in my scenario I use Policy Based forwarding to send 'default' traffic via the preferred ISP with a Monitor so if the interface/path goes down the PDF policy is disabled and then traffic follows the virtual router and that 'default' path is ISP B.

Hope that makes sense.

[Andreikin]

@OtakarKlier yes, that's the difference - I am trying to do something with traffic sourced from one of the interfaces of firewall, while PBF for such scenario will not work unfortunately 

[OtakarKlier]

Hello,

I havent tested is, but I think you can since you can select the source IP (external VPN IP) and then setup the forwarder?

Just a thought to test?

[Andreikin]

@OtakarKlier Hi, according to this document it won't work for IPSEC tunnels:

Policy-based forwarding doesn't work for traffic sourced from t... - Knowledge Base - Palo Alto Netw...

[OtakarKlier]

Sorry I couldnt be more help.

[reaper]

i think this is expected behavior in this design

without ECMP the system relies on regular routing to decide which interface packets need to egress out of, so in case of an ISP outage ECMP will no longer be able to 'force' the egress interface

in cases where tunnel redundancy is most important, I usually create 2 Virtual routers and attach each ISP to it's own VR. that ensures both tunnels are up and running without any (potential) conflicts

I can then use PBF and regular routing to control where client traffic is sent and failover where needed 

[TomYoung]

Hi @OtakarKlier and @Andreikin ,

Sorry for the late reply!  The original design is good.

I would create specific security policy rules (both ways) to block traffic from one public IP to egress on the other interface.  It sounds like both ISPs are in the same zone, and the traffic is allowed via the intrazone-default rule.

Thanks,

Tom