IPSec VPN tunnel no longer working

[oschuler]

Hello guys

We have a few VPN tunnels between our PA-2050 (in HA cluster) and some WatchGuard firewalls (different models). We migrated these tunnels to the PA-2050 a few weeks ago and they ran stable. Now suddenly two of 10 tunnels are down and we don't get them back up.

Here's what we tried so far:

- Rebooting the WatchGuard firewalls

- Suspending the active PA-2050 so the standby HA device takes over

- Removing the VPN config on the WatchGuards and rebuild them (only VPN part)

- Overwrite the PSK on both ends

- On the PA-2050 CLI: clear vpn ike-sa gateway <gw-name> and clear vpn ipsec-sa tunnel <tunnel-name>

- some more small stuff

The config is the same on all tunnels, including the two not working...

Any inputs would on how we can further troubleshoot the issue would be much appreciated. The connection by the way is working. We can ping the endpoints mutually and we can also access the external admin interface of the remote WatchGuard firewalls...

Thanks!

Oliver

[Ameya-Kawimandan · Accepted solution]

Oliver,

In your case it was the combination of downgrading and clearing the ike session in Discard state.

Tunnels did not come up after downgrade as their was a stale ike session  on the firewall.

Typically in this case ike-manager logs would show  message "Phase-2 rekey request ignored: previous request still in progress"

[Ameya-Kawimandan]

Could you try downgrading to Content version : 315-1427

Saw couple of cases where this worked for  me.

Regards,

Ameya

[oschuler]

We managed to get one of the two tunnels up by changing the external IP address of the tunnel endpoint on the PA-2050 side. We have multiple external IP addresses and all tunnels were configured to use the same IP. At the time we changed the IP on one of the tunnels not working it came back up... We tried the same for the other tunnel still down and it didn't help...

None of the reverts back to any of the following versions worked for us:

- 316-1432

- 315-1427

- 314-1424

For the last two (315 and 314) we could only install the Content package using Panorama. It was not possible to also install the App package of that version (error message we got was: No matching contents package found in panupv2-all-apps-<version>).

Thanks,

Oliver

[oschuler]

After countless trials we finally got the last tunnel up as well. It came back up when we cleared the IPSec/IKE session for that tunnel on the PA-2050.

[mikand]

A longshot here but did these problems start last sunday (1st july)?

Im thinking of the leapsecond stuff...

[jthompson1]

Having the same issue, VPN tunnels are not working to a Fortigate 310B. Rebooting the PA-500 solves it temporarily and then it dies again a day later. Clearing out the VPN IKE, IPSEC, and flow does not work.

I just keep getting IKE timeouts. I will try to revert content but I have a feeling it might be a software issue. Currently running 4.1.6. Any known issues with this?

-J

[oschuler]

We're running OS 4.1.6 as well.

It might be that the issue has been fixed on our side by reverting back to Content version older than 316-1432 as Ameya suggested. We tested so many different settings to get the tunnel up that we're now unable to tell what setting(s) fixed it in the end...

[Ameya-Kawimandan · Accepted solution]

Oliver,

In your case it was the combination of downgrading and clearing the ike session in Discard state.

Tunnels did not come up after downgrade as their was a stale ike session  on the firewall.

Typically in this case ike-manager logs would show  message "Phase-2 rekey request ignored: previous request still in progress"

[jthompson1]

I've downgraded to 4.1.4. The content change didn't work. We'll see how long the tunnel stays up.

-J

[oschuler]

Jason, I think Ameya is totally right. Reverting back to an older apps/content version alone didn't work for me. I had to explicitly close the stale IKE sessions using the Session Browser. I guess your OS downgrade worked because your box restarted after the downgrade which also clears out all stale sessions...

OS version 4.1.6 works great for us.