IPSec VPN tunnel no longer working

[oschuler]

Hello guys

We have a few VPN tunnels between our PA-2050 (in HA cluster) and some WatchGuard firewalls (different models). We migrated these tunnels to the PA-2050 a few weeks ago and they ran stable. Now suddenly two of 10 tunnels are down and we don't get them back up.

Here's what we tried so far:

- Rebooting the WatchGuard firewalls

- Suspending the active PA-2050 so the standby HA device takes over

- Removing the VPN config on the WatchGuards and rebuild them (only VPN part)

- Overwrite the PSK on both ends

- On the PA-2050 CLI: clear vpn ike-sa gateway <gw-name> and clear vpn ipsec-sa tunnel <tunnel-name>

- some more small stuff

The config is the same on all tunnels, including the two not working...

Any inputs would on how we can further troubleshoot the issue would be much appreciated. The connection by the way is working. We can ping the endpoints mutually and we can also access the external admin interface of the remote WatchGuard firewalls...

Thanks!

Oliver

[UhMayYeah]

Could you try downgrading to Content version : 315-1427

Saw couple of cases where this worked for  me.

Regards,

Ameya

[oschuler]

We managed to get one of the two tunnels up by changing the external IP address of the tunnel endpoint on the PA-2050 side. We have multiple external IP addresses and all tunnels were configured to use the same IP. At the time we changed the IP on one of the tunnels not working it came back up... We tried the same for the other tunnel still down and it didn't help...

None of the reverts back to any of the following versions worked for us:

- 316-1432

- 315-1427

- 314-1424

For the last two (315 and 314) we could only install the Content package using Panorama. It was not possible to also install the App package of that version (error message we got was: No matching contents package found in panupv2-all-apps-<version>).

Thanks,

Oliver

[oschuler]

After countless trials we finally got the last tunnel up as well. It came back up when we cleared the IPSec/IKE session for that tunnel on the PA-2050.

[mikand]

A longshot here but did these problems start last sunday (1st july)?

Im thinking of the leapsecond stuff...

[jthompson1]

Having the same issue, VPN tunnels are not working to a Fortigate 310B. Rebooting the PA-500 solves it temporarily and then it dies again a day later. Clearing out the VPN IKE, IPSEC, and flow does not work.

I just keep getting IKE timeouts. I will try to revert content but I have a feeling it might be a software issue. Currently running 4.1.6. Any known issues with this?

-J

[oschuler]

We're running OS 4.1.6 as well.

It might be that the issue has been fixed on our side by reverting back to Content version older than 316-1432 as Ameya suggested. We tested so many different settings to get the tunnel up that we're now unable to tell what setting(s) fixed it in the end...

[UhMayYeah · Accepted solution]

Oliver,

In your case it was the combination of downgrading and clearing the ike session in Discard state.

Tunnels did not come up after downgrade as their was a stale ike session  on the firewall.

Typically in this case ike-manager logs would show  message "Phase-2 rekey request ignored: previous request still in progress"

[jthompson1]

I've downgraded to 4.1.4. The content change didn't work. We'll see how long the tunnel stays up.

-J

[oschuler]

Jason, I think Ameya is totally right. Reverting back to an older apps/content version alone didn't work for me. I had to explicitly close the stale IKE sessions using the Session Browser. I guess your OS downgrade worked because your box restarted after the downgrade which also clears out all stale sessions...

OS version 4.1.6 works great for us.

[jthompson1]

Yeah I'll have to see... problem is I can reboot and it just comes back. I will try other methods of clearing out the stale sessions but it's still a problem Palo Alto has to address. When a session ends a tunnel should re-establish and it doesn't, instead it just hangs. Not good. I'll update this thread if I find anything else.

-J

[User_333]

If your side gets this "timeouts" log entries it would be interesting to see what the logs on the other side are (if its possible to get them).

[Support_CC]

For our client problems started with

"Application Identification package upgraded from version 316-1432 to 317-1438 by Auto update agent"

and the ike traffic was sometimes recognized or maybe not recognized as "unknown-udp".

I added a rule with service to allow udp:500 between VPN peers as a quick fix which seems to work so far.

You might also want to verify ike sessions (show session all filter destination-port 500) and do some cleanup if they are in discard state.

[pg_itdept]

We suddenly started getting in the last couple of weeks.  Our VPN had been solid as rock for a few months and then after an issue with the routing along the way (our ISP) the VPN refused to recover unless we rebooted.  It happened again over the weekend.  I am not aware of any issues with the ISP this time however we had to reboot our PA-500 (4.1.5) to get the connection back.

Will see if next time this occurs I can clear out the stale sessions

[oschuler]

We noticed that it's somtimes necessary to not only clear the stale session but also to clear the vpn flow of that tunnel:

1. clear session id <value>

2. clear vpn flow tunnel-id <value>