PAN site to site VPN to AWS

[SamuelCardoz]

We had a site to sit VPN between on premise PAN going to AWS.

The tunnel was established and does not show any downtime but the issue we encounter is that when the Tunnel Monitor IP(169.254.2.x/30) and (169.254.3.x/30) is not pingable/unreachable PAN will remove the route going to AWS in result we are not able to connect to the AWS LAN segment.

As a workaround we need to restart the tunnel and after restart we are able to restore the connection.

-No logs about VPN tunnel down

-On the routed logs it shows that route has been remove it occur 6 to 8 times a day.

-No DPD logs cause downtime

-No issue on rekey
Sample logs

2021-12-06 10:09:41.324 +0800 MON: status update md(135: 169.254.2.10 => 169.254.2.9 => 169.254.2.9) Failed
2021-12-06 10:09:41.324 +0800 MON: status update monitor(vr GABC_VR: 10.x.x.x > 0.0.0.0) Down
2021-12-06 10:09:41.325 +0800 MON: status update monitor(vr GABC_VR: 10.x.x.x > 0.0.0.0) Down
2021-12-06 10:09:41.325 +0800 MON: status update monitor(vr GABC_VR: 10.x.x.x > 0.0.0.0) Down
2021-12-06 10:13:32.324 +0800 MON: status update md(136: 169.254.3.12 => 169.254.3.11 => 169.254.3.11) Failed
2021-12-06 10:13:32.324 +0800 MON: status update monitor(vr GABC_VR: 10.x.x.x > 0.0.0.0) Down
2021-12-06 10:13:32.325 +0800 MON: status update monitor(vr GABC_VR: 10.x.x.x > 0.0.0.0) Down
2021-12-06 10:13:32.325 +0800 MON: status update monitor(vr GABC_VR: 10.x.x.x > 0.0.0.0) Down
Any idea what is causing on this? 

[OtakarKlier]

Hello,

Is there traffic constantly flowing over the tunnels? Do you have a policy that prefers one tunnel over the others? When i find tunnel issues to AWS, its always AWS because like you, I cant find anything wrong on the PAN side and you cant see what is happening on the AWS side :(.

Regards,

[BPry]

@SamuelCardoz,

As @OtakarKlier already mentioned, this is almost certainly something on the AWS side of things. When your tunnel monitoring fails, do you have any other monitoring configured to resources on the other side of the tunnel? Essentially, it's always helpful to see if something else is seeing the outage at the same time that your firewall is, or if its only after the firewall takes the tunnel down following the tunnel monitoring failure.

Additionally when your talking about an AWS tunnel I actually don't like using the tunnel interfaces for monitoring. I'll have the PAN monitor a resource in AWS instead. 

[SamuelCardoz]

Hello Otakarklier

Thanks for the reply And to answer your query it has no traffic passing to the tunnel once the two monitor IP is not reachable since the route going to the VPN tunnel has been remove. 

To add more info AWS provided as two tunnel to configure on our end that's why we have two monitor IP's. 

Thanks

[SamuelCardoz]

Hello BPry

Thanks for the response. We did not see any tunnel down the only thing is that when the two monitor IP is not reachable the route will be remove from the routing table and the result is we could not reach the network segment on AWS.
Is it possible to change the monitor IP's to an IP address when in the AWS LAN segment like a EC2 server or a workstation?

Is there a way on PAN that if ever the monitor IP is down it will not remove the route on the routing table? or it is the default settings on PAN that we could not change?

May i know if what is the possible action i take to test and would result to resolve the issue.

Thanks

[aleksandar.astardzhiev]

Hi @SamuelCardoz ,

- Yes, you can configure any IP address for tunnel monitor, as long as it is reachable through the tunnel. In most cases it is preferable to use IP address assigned to network device, as this will assure that it is always up (so the tunnel will not go down if the end host is shutdown or during maintenance. But again it is completely up to you to decide which IP to use

- The whole purpose of the tunnel monitor is to "disable" the static route from the routing table, so the firewall can fallback to alternative path, if this tunnel is down. So if you want firewall to not remove the route from routing table if the tunnel is down, you should simply disable the tunnel monitor. But if you do that, firewall will not have a way to failover between the tunnels to AWS

[OtakarKlier]

Hello,

What I do is use policy based forwarding to send all traffic down one tunnel and then use the other tunnel as backup. If no traffic is flowing over it, then this could be why it goes down periodically. I would say even a simple dead peer detection policy so the PAN sends a ping down the tunnel now and then should keep it up.

Regards,

[SamuelCardoz]

Hello @aleksandar.astardzhiev 
 

The tunnel monitoring i mentioned is this under Path monitoring because on Path monitoring i could not change to any IP except the IP address that is on /30.
Please see picture below if this tunnel monitoring is down it will remove the route in the routing table.

I tried to disable this one/remove it show intermittent connection on the VPN tunnel.

Thanks

[aleksandar.astardzhiev]

Hi @SamuelCardoz ,

I don't remember personally use path monitor option under static route definition, but I am almost certain that you can use any destination address, it doesn't have to be from the same directly connect network as the outbound interface. Path monitor is similar to "IP SLA" with over network vendors, the purpose is to send probes to address reachable over that path, it should be up to you to decide how far you will test the path (is it up to next hop, or all the way to the destination). If it does not work with other address, I believe the problem is with IPsec phase2 proxy-id (phase-selectors/encryption domains), which don't match the traffic with that source and destination.

As a second thought...This probably could be AWS limitation - From your screenshot I can see that you are using th 169.254.x.x reserved range for the point-to-point connection between the IPsec peers (probably AWS requires you to use that?). Since path monitor and tunnel monitor will always use the IP addresses assigned on the egress interface, your probes will always be sourced from 169.254.x.x, which I am not sure if it routable in your VPC - there for if you try to monitor anything else in AWS, the return traffic will not be routed back and monitor will fail. But this is purely public cloud limitation

Disabling path/tunnel monitor, shouldn't affect your IPsec tunnel. If you think about it, it is a method for the FW to dynamically detect issue with the tunnel and automatically switch to backup path. The only way disabling path/tunnel monitor could cause intermittent issues is if have equal cost routes at any side of the tunnel

- either the FW is sending traffic through both tunnels

- or AWS is returning the traffic through both tunnels, causing again asymetric routing