topic Re: IPSec VPN tunnel no longer working in General Topics

IPSec VPN tunnel no longer working

oschuler — Thu, 05 Jul 2012 16:47:55 GMT

Hello guys

We have a few VPN tunnels between our PA-2050 (in HA cluster) and some WatchGuard firewalls (different models). We migrated these tunnels to the PA-2050 a few weeks ago and they ran stable. Now suddenly two of 10 tunnels are down and we don't get them back up.

Here's what we tried so far:

- Rebooting the WatchGuard firewalls

- Suspending the active PA-2050 so the standby HA device takes over

- Removing the VPN config on the WatchGuards and rebuild them (only VPN part)

- Overwrite the PSK on both ends

- On the PA-2050 CLI: clear vpn ike-sa gateway <gw-name> and clear vpn ipsec-sa tunnel <tunnel-name>

- some more small stuff

The config is the same on all tunnels, including the two not working...

Any inputs would on how we can further troubleshoot the issue would be much appreciated. The connection by the way is working. We can ping the endpoints mutually and we can also access the external admin interface of the remote WatchGuard firewalls...

Thanks!

Oliver

Re: IPSec VPN tunnel no longer working

UhMayYeah — Thu, 05 Jul 2012 18:42:52 GMT

Could you try downgrading to Content version : 315-1427

Saw couple of cases where this worked for me.

Regards,

Ameya

Re: IPSec VPN tunnel no longer working

oschuler — Thu, 05 Jul 2012 20:12:01 GMT

We managed to get one of the two tunnels up by changing the external IP address of the tunnel endpoint on the PA-2050 side. We have multiple external IP addresses and all tunnels were configured to use the same IP. At the time we changed the IP on one of the tunnels not working it came back up... We tried the same for the other tunnel still down and it didn't help...

None of the reverts back to any of the following versions worked for us:

- 316-1432

- 315-1427

- 314-1424

For the last two (315 and 314) we could only install the Content package using Panorama. It was not possible to also install the App package of that version (error message we got was: No matching contents package found in panupv2-all-apps-<version>).

Thanks,

Oliver

Re: IPSec VPN tunnel no longer working

oschuler — Thu, 05 Jul 2012 21:14:22 GMT

After countless trials we finally got the last tunnel up as well. It came back up when we cleared the IPSec/IKE session for that tunnel on the PA-2050.

Re: IPSec VPN tunnel no longer working

mikand — Fri, 06 Jul 2012 12:21:46 GMT

A longshot here but did these problems start last sunday (1st july)?

Im thinking of the leapsecond stuff...

Re: IPSec VPN tunnel no longer working

jthompson1 — Fri, 06 Jul 2012 12:40:53 GMT

Having the same issue, VPN tunnels are not working to a Fortigate 310B. Rebooting the PA-500 solves it temporarily and then it dies again a day later. Clearing out the VPN IKE, IPSEC, and flow does not work.

I just keep getting IKE timeouts. I will try to revert content but I have a feeling it might be a software issue. Currently running 4.1.6. Any known issues with this?

-J

Re: IPSec VPN tunnel no longer working

oschuler — Fri, 06 Jul 2012 15:46:02 GMT

We're running OS 4.1.6 as well.

It might be that the issue has been fixed on our side by reverting back to Content version older than 316-1432 as Ameya suggested. We tested so many different settings to get the tunnel up that we're now unable to tell what setting(s) fixed it in the end...

Re: IPSec VPN tunnel no longer working

UhMayYeah — Fri, 06 Jul 2012 17:41:57 GMT

Oliver,

In your case it was the combination of downgrading and clearing the ike session in Discard state.

Tunnels did not come up after downgrade as their was a stale ike session on the firewall.

Typically in this case ike-manager logs would show message "Phase-2 rekey request ignored: previous request still in progress"

Re: IPSec VPN tunnel no longer working

jthompson1 — Fri, 06 Jul 2012 17:45:18 GMT

I've downgraded to 4.1.4. The content change didn't work. We'll see how long the tunnel stays up.

-J

Re: IPSec VPN tunnel no longer working

oschuler — Fri, 06 Jul 2012 18:43:26 GMT

Jason, I think Ameya is totally right. Reverting back to an older apps/content version alone didn't work for me. I had to explicitly close the stale IKE sessions using the Session Browser. I guess your OS downgrade worked because your box restarted after the downgrade which also clears out all stale sessions...

OS version 4.1.6 works great for us.

Re: IPSec VPN tunnel no longer working

jthompson1 — Fri, 06 Jul 2012 19:26:55 GMT

Yeah I'll have to see... problem is I can reboot and it just comes back. I will try other methods of clearing out the stale sessions but it's still a problem Palo Alto has to address. When a session ends a tunnel should re-establish and it doesn't, instead it just hangs. Not good. I'll update this thread if I find anything else.

-J

Re: IPSec VPN tunnel no longer working

User_333 — Fri, 06 Jul 2012 21:46:53 GMT

If your side gets this "timeouts" log entries it would be interesting to see what the logs on the other side are (if its possible to get them).

Re: IPSec VPN tunnel no longer working

Support_CC — Sat, 07 Jul 2012 15:26:44 GMT

For our client problems started with

"Application Identification package upgraded from version 316-1432 to 317-1438 by Auto update agent"

and the ike traffic was sometimes recognized or maybe not recognized as "unknown-udp".

I added a rule with service to allow udp:500 between VPN peers as a quick fix which seems to work so far.

You might also want to verify ike sessions (show session all filter destination-port 500) and do some cleanup if they are in discard state.

Re: IPSec VPN tunnel no longer working

pg_itdept — Mon, 09 Jul 2012 07:44:47 GMT

We suddenly started getting in the last couple of weeks. Our VPN had been solid as rock for a few months and then after an issue with the routing along the way (our ISP) the VPN refused to recover unless we rebooted. It happened again over the weekend. I am not aware of any issues with the ISP this time however we had to reboot our PA-500 (4.1.5) to get the connection back.

Will see if next time this occurs I can clear out the stale sessions

Re: IPSec VPN tunnel no longer working

oschuler — Mon, 09 Jul 2012 07:53:02 GMT

We noticed that it's somtimes necessary to not only clear the stale session but also to clear the vpn flow of that tunnel:

1. clear session id <value>

2. clear vpn flow tunnel-id <value>

Re: IPSec VPN tunnel no longer working

pg_itdept — Mon, 09 Jul 2012 08:00:25 GMT

Thank you for that oschuler will try it next time

Re: IPSec VPN tunnel no longer working

oschuler — Wed, 11 Jul 2012 06:11:55 GMT

Looks like the latest release (version 318) contains modifications on the 'ike' decoder. We'll install it on Sunday, hopefully the issues are gone with it...

Re: IPSec VPN tunnel no longer working

pg_itdept — Wed, 11 Jul 2012 09:59:19 GMT

OK our VPN is lasting for 28 hours with an 8 hour lifetime.

Having just got it back, it would appear that the UDP packets are not being correctly identified as IKE and being dropped at the firewall. I am guessing this is on a re-negotiation of the link. I just looked for 318 and there was no sign of it so I will keep an eye on it being available

Re: IPSec VPN tunnel no longer working

jthompson1 — Thu, 12 Jul 2012 13:53:17 GMT

I haven't had an issue since reverting back to 4.1.4. Perhaps there was just an issue with the install...

When 4.1.7 is released I'll try that along with the 318 content.

-J