2xISPs and 2 VPN tunnels - tunnel failover issue

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

2xISPs and 2 VPN tunnels - tunnel failover issue

L2 Linker

Hello Team,

 

I am running into an issue with our setup. We have single PA460 box connected to 2 ISPs same time, i.e. Ethernet1/3 is to ISP1 and Ethernet1/4 to ISP2. We are running 2 default routes setup like this with ECMP enabled, so traffic is been load-balanced betwen 2

ISPs with a hash based on source. We also have 'strict source path' option enabled in ECMP settings. Both default routes have monitor enabled, so if for example Google DNS IP is not reachable over this ISP, it will be removed from routing table. On top of that we are running 2 VPN tunnels from this box, each of those using one of the physical interface IP address bellow as source. Both VPN tunnels are been terminated on the same remote Public IP. 

 

default_routes.jpg

 

 

 

 

 

 

 

 

 

 

 

 

When 2 ISPs are running normally and traffic is going through those, with 'strict source path' option we have VPN tunnel traffic bound to the same VPN tunnels they are sourced from. So encrypted traffic for VPN1 is always going out through Etherent1/3 and encrypted traffic for VPN2 is going through Ethernet1/4. 

 

But, for example, if ISP1 is DOWN, we see that default route through ISP1 via Ethernet1/3 is down and removed from routing table, which is expected. What we did not expect is we see in traffic captures that Palo appliance is trying to bring up VPN1 through Ethernet1/4, while still using public IP of Ethernet1/3. That VPN is not obviously coming up as public IP of Ethernet1/3 is not reachable externally any longer, but an actual behavior is concerning and in few corner cases might lead to a weird case that VPN1 will be established using Ethernet1/4 as egress interface. I've tried to use VPN monitoring profile with 'wait-recover' option, but it seems to lead to the same issue. 

 

Is there any hints or pointers you can give me in this situation and how to avoid it?

 

Thanks!

 

 

11 REPLIES 11

Cyber Elite
Cyber Elite

Hello,

Do the VPN tunnels terminate at the same endpoint device, on the other end?

Regards,

L2 Linker

Hi @OtakarKlier from our side and from remote side they do terminate on a very same device. We just have two tunnels going over 2 ISPs for a redundancy purposes. 

Cyber Elite
Cyber Elite

Hello,

I'm gonna guess that this is because of the default routes and IPSec tunnel config. The tunnel is trying to use the IP of the down interface via the up interface. Cant think of a way to prevent the IPSec tunnel with IP of the down interface to not use the up interface etc. I dont run ECMP so cant say I've run into this. I usually just use OSPF routing for the tunnels and for the ISP use Policy Based Forwarding to prefer one over the other, so just failover.

Hopefully someone else can jump in and share their thoughts. @BPry @reaper @aleksandar.astardzhiev @TomYoung 

L2 Linker

@OtakarKlier yes, that happens indeed because of the fact, that 'strict source path' thingie only works when you have ECMP in effect, which is when you have at least 2 routes to the same destination and same metric. When one of the ISPs is down, than 'strict source path' is no longer in effect I guess.

 

But to your answer, can you elaborate what are you using PBF for? AFAIK you can only force traffic THROUGH firewall to be affected by PBF and not traffic source/destined to it. 

Cyber Elite
Cyber Elite

Hello,

Policy based forwarding take effect prior to the virtual router. So in my scenario I use Policy Based forwarding to send 'default' traffic via the preferred ISP with a Monitor so if the interface/path goes down the PDF policy is disabled and then traffic follows the virtual router and that 'default' path is ISP B.

 

Hope that makes sense.

@OtakarKlier yes, that's the difference - I am trying to do something with traffic sourced from one of the interfaces of firewall, while PBF for such scenario will not work unfortunately 

Cyber Elite
Cyber Elite

Hello,

I havent tested is, but I think you can since you can select the source IP (external VPN IP) and then setup the forwarder?

OtakarKlier_0-1706636607290.png

OtakarKlier_1-1706636629979.png

 

Just a thought to test?

L2 Linker

Cyber Elite
Cyber Elite

Sorry I couldnt be more help.

Cyber Elite
Cyber Elite

i think this is expected behavior in this design

without ECMP the system relies on regular routing to decide which interface packets need to egress out of, so in case of an ISP outage ECMP will no longer be able to 'force' the egress interface

 

in cases where tunnel redundancy is most important, I usually create 2 Virtual routers and attach each ISP to it's own VR. that ensures both tunnels are up and running without any (potential) conflicts

I can then use PBF and regular routing to control where client traffic is sent and failover where needed 

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

Hi @OtakarKlier and @Andreikin ,

 

Sorry for the late reply!  The original design is good.

 

I would create specific security policy rules (both ways) to block traffic from one public IP to egress on the other interface.  It sounds like both ISPs are in the same zone, and the traffic is allowed via the intrazone-default rule.

 

Thanks,

 

Tom

Help the community: Like helpful comments and mark solutions.
  • 1383 Views
  • 11 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!