topic Re: IPSec VPN restarts very often in General Topics

IPSec VPN restarts very often

chmielniak — Mon, 24 Sep 2012 10:52:58 GMT

Hallo,

I have defined a IPSec VPN connection with following params:

ike: 3des/sha1/dh5 Lifetime: 8 hours

ipsec: ESP/3des/sha1/dh5 Lifetime: 30 minutes (life size not set, shows 0MB)

ike gateway: main mode, DP enabled

The connection is established but in system log I see very often (every 5 sec.) tunnel is again and again down and up. We have packet lost about 0.5%.

Any ideas? I've already configured the connection from scratch again.

Jacek.

Log file:

2012/09/24 12:36:39 ipsec-key-delete IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8C5FC8B5/0xFFFD0AD9.

2012/09/24 12:36:39 ike-send-p2-delete IKE protocol IPSec SA delete message sent to peer. SPI:0x8C5FC8B5.

2012/09/24 12:36:38 ipsec-key-install IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xDF1F9E37/0xFFFD0ADA lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:38 ike-nego-p2-succ IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xCFE39FEB, SPI:0xDF1F9E37/0xFFFD0ADA.

2012/09/24 12:36:38 ike-nego-p2-start IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xCFE39FEB.

2012/09/24 12:36:35 ipsec-key-delete IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xCDCD7E83/0xFFFD0AD8.

2012/09/24 12:36:35 ike-send-p2-delete IKE protocol IPSec SA delete message sent to peer. SPI:0xCDCD7E83.

2012/09/24 12:36:34 ipsec-key-install IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8C5FC8B5/0xFFFD0AD9 lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:34 ike-nego-p2-succ IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x756F7417, SPI:0x8C5FC8B5/0xFFFD0AD9.

2012/09/24 12:36:34 ike-nego-p2-start IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x756F7417.

2012/09/24 12:36:31 ipsec-key-delete IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xE36D50CD/0xFFFD0AD7.

2012/09/24 12:36:31 ike-send-p2-delete IKE protocol IPSec SA delete message sent to peer. SPI:0xE36D50CD.

2012/09/24 12:36:30 ipsec-key-install IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xCDCD7E83/0xFFFD0AD8 lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:30 ike-nego-p2-succ IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x43C3E41C, SPI:0xCDCD7E83/0xFFFD0AD8.

2012/09/24 12:36:30 ike-nego-p2-start IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x43C3E41C.

2012/09/24 12:36:27 ipsec-key-delete IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8D0BBED9/0xFFFD0AD6.

2012/09/24 12:36:27 ike-send-p2-delete IKE protocol IPSec SA delete message sent to peer. SPI:0x8D0BBED9.

2012/09/24 12:36:26 ipsec-key-install IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xE36D50CD/0xFFFD0AD7 lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:26 ike-nego-p2-succ IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x15CF19C6, SPI:0xE36D50CD/0xFFFD0AD7.

2012/09/24 12:36:26 ike-nego-p2-start IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x15CF19C6.

Re: IPSec VPN restarts very often

mikand — Tue, 25 Sep 2012 03:58:00 GMT

Since it takes two to tango VPN - what do you have at the other side?

And if possible, could you put one of those peers much closer to your PA to rule out any interference from the network(s) in between?

Also, how is your security policies setup for this traffic?

And is your ipsec setup on a physical interface (which perhaps goes up and down?) or a loopback interface?

Re: IPSec VPN restarts very often

chmielniak — Tue, 25 Sep 2012 07:20:15 GMT

I have 3 VPNs running. The problem appeared after some update in 4.1.x with all 3 connections. I can control both sides of only one connection (it's cisco router IPSec with VTI). I've just deleted the configuration on both sides and recreated it with the same parameters and it works now.

The other two VPN partners haven't change a thing and it shows above problem.

All IPSec connections are setup on the same physical interface together with normal internet traffic. I observe no problems with the interface. Security policies allow ssh connections outgoing.

How can I see what makes the connections up and down?

Re: IPSec VPN restarts very often

ppatel — Tue, 25 Sep 2012 08:10:20 GMT

Make sure the Crypto settings are same on both the sides and try initiating the tunnel traffic from the remote side.

Also try configuring the ipsec-crypto to DH group to "no-pfs" on both the sides. Clear the VPN tunnels on the Palo Alto side.

admin@PA> clear vpn ike-sa gateway

<value> clear for given IKE gateway

admin@PA> clear vpn ipsec-sa tunnel

<value> clear for given VPN tunnel

Try initiate the tunnel from the cisco side.

Monitor the system logs on the firewall to see the IPSEC negotiation. Check to see if the tunnel comes up.

Which version of software are you using on the firewall? There is an issue with software release 4.1.5 where after an upgrade or intermittently IPSec VPN tunnel would not come up when Palo Alto Networks firewall initiates a

connection to a Cisco ASA device. [Bug # 39884] This issue was addressed in S/w 4.1.6.

Let me know if this helps.

Regards

Parth

Re: IPSec VPN restarts very often

chmielniak — Tue, 25 Sep 2012 08:38:01 GMT

We have 4.1.7.

I cannot control the other side of tunnel. Those are partners, that request configuration according to their policy. I have checked the ipsec params requested. They are set correct on both sides. IPSec tunnel is established, but in log i can see those IKE deletes every ca. 10 sec. I had to disable information log on syslog.

One on VPNS ends on Fortinet. The only one VPN, that is working has no Proxy-ID defined. The other two with problems have Proxy Ids defined in IPSec tunnel.

I can see any reason for "IKE protocol IPSec SA delete message sent to peer". We experience long RTT and packet lost on those connections.

Jacek.

Re: IPSec VPN restarts very often

ppatel — Tue, 25 Sep 2012 08:53:03 GMT

Try to have a look at the ikemgr log on the firewall.

admin@PA> tail follow yes mp-log ikemgr.log

The above log will be able to give detailed logging on the ike negotiation process that is done by the firewall

Try to clear the vpn flow using the commands that were given before and run the tail command.

Please open a support ticket with TAC to troubleshoot the issue if it is still not resolving. It would be great to have access to both the sides of the tunnel to debug/troubleshoot Ipsec.

Regards

Parth

Re: IPSec VPN restarts very often

chmielniak — Tue, 25 Sep 2012 10:16:00 GMT

I have disabled one vpn, so that I could cut the debug log for only one VPN. I set level with: debug ike global on normal
I have resetted the connection. I have cut three phases delete/establish: (XX.XX.XX.XX is own IP, YY.YY.YY.YY is partner IP)

2012-09-25 12:00:47 [INFO]: panike_keyacquire_cb 2fe29fa8

2012-09-25 12:00:47 [INFO]: IPsec-SA request for YY.YY.YY.YY queued since no phase1 found

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-1 NEGOTIATION STARTED AS INITIATOR, MAIN MODE <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] cookie:6041d73ad3ab677f:0000000000000000 <====

2012-09-25 12:00:47 [INFO]: received Vendor ID: DPD

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-1 NEGOTIATION SUCCEEDED AS INITIATOR, MAIN MODE <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] cookie:6041d73ad3ab677f:90e8c84f5cf54d25 lifetime 28800 Sec <====

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION STARTED AS INITIATOR, (QUICK MODE) <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x9395643A <====

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION SUCCEEDED AS INITIATOR, (QUICK MODE) <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x9395643A, SPI:0xC1B84E56/0xFFFD4D90 <====

2012-09-25 12:00:47 [INFO]: SADB_UPDATE ul_proto=255 src=YY.YY.YY.YY[500] dst=XX.XX.XX.XX[500] satype=ESP samode=tunl spi=0xC1B84E56 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:47 [INFO]: SADB_ADD ul_proto=255 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP samode=tunl spi=0xFFFD4D90 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:47 [INFO]: IPsec-SA established: ESP/Tunnel YY.YY.YY.YY[500]->XX.XX.XX.XX[500] spi=3250081366(0xc1b84e56)

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> IPSEC KEY INSTALLATION SUCCEEDED <====

====> Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xC1B84E56/0xFFFD4D90 lifetime 1800 Sec lifesize unlimited <====

2012-09-25 12:00:47 [INFO]: keymirror add start ++++++++++++++++

2012-09-25 12:00:47 [INFO]: keymirror add for gw 4, tn 8, selfSPI C1B84E56, retcode 0.

2012-09-25 12:00:47 [INFO]: keymirror del start ----------------

2012-09-25 12:00:47 [INFO]: keymirror del for gw 4, tn 8, selfSPI F703F0F5, retcode 0.

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> IPSEC KEY DELETED <====

====> Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xF703F0F5/0xFFFD4D8E <====

2012-09-25 12:00:47 [INFO]: SADB_DELETE ul_proto=0 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP spi=0xF703F0F5

2012-09-25 12:00:47 [INFO]: received PFKEY_DELETE seq=0 satype=ESP spi=0xF703F0F5

2012-09-25 12:00:54 [INFO]: panike_keyacquire_cb 2fe2ad98

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION STARTED AS INITIATOR, (QUICK MODE) <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x173FBEFA <====

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION SUCCEEDED AS INITIATOR, (QUICK MODE) <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x173FBEFA, SPI:0x8605AB62/0xFFFD4D91 <====

2012-09-25 12:00:54 [INFO]: SADB_UPDATE ul_proto=255 src=YY.YY.YY.YY[500] dst=XX.XX.XX.XX[500] satype=ESP samode=tunl spi=0x8605AB62 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:54 [INFO]: SADB_ADD ul_proto=255 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP samode=tunl spi=0xFFFD4D91 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:54 [INFO]: IPsec-SA established: ESP/Tunnel YY.YY.YY.YY[500]->XX.XX.XX.XX[500] spi=2248518498(0x8605ab62)

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> IPSEC KEY INSTALLATION SUCCEEDED <====

====> Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8605AB62/0xFFFD4D91 lifetime 1800 Sec lifesize unlimited <====

2012-09-25 12:00:54 [INFO]: keymirror add start ++++++++++++++++

2012-09-25 12:00:54 [INFO]: keymirror add for gw 4, tn 8, selfSPI 8605AB62, retcode 0.

2012-09-25 12:00:54 [INFO]: keymirror del start ----------------

2012-09-25 12:00:54 [INFO]: keymirror del for gw 4, tn 8, selfSPI C1B84E56, retcode 0.

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> IPSEC KEY DELETED <====

====> Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xC1B84E56/0xFFFD4D90 <====

2012-09-25 12:00:54 [INFO]: SADB_DELETE ul_proto=0 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP spi=0xC1B84E56

2012-09-25 12:00:54 [INFO]: received PFKEY_DELETE seq=0 satype=ESP spi=0xC1B84E56

2012-09-25 12:00:58 [INFO]: panike_keyacquire_cb 2fe29fa8

2012-09-25 12:00:58 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION STARTED AS INITIATOR, (QUICK MODE) <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xFB43C451 <====

2012-09-25 12:00:58 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION SUCCEEDED AS INITIATOR, (QUICK MODE) <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xFB43C451, SPI:0x9838E09A/0xFFFD4D92 <====

2012-09-25 12:00:58 [INFO]: SADB_UPDATE ul_proto=255 src=YY.YY.YY.YY[500] dst=XX.XX.XX.XX[500] satype=ESP samode=tunl spi=0x9838E09A authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:58 [INFO]: SADB_ADD ul_proto=255 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP samode=tunl spi=0xFFFD4D92 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:58 [INFO]: IPsec-SA established: ESP/Tunnel YY.YY.YY.YY[500]->XX.XX.XX.XX[500] spi=2553864346(0x9838e09a)

2012-09-25 12:00:58 [PROTO_NOTIFY]: ====> IPSEC KEY INSTALLATION SUCCEEDED <====

====> Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x9838E09A/0xFFFD4D92 lifetime 1800 Sec lifesize unlimited <====

2012-09-25 12:00:58 [INFO]: keymirror add start ++++++++++++++++

2012-09-25 12:00:58 [INFO]: keymirror add for gw 4, tn 8, selfSPI 9838E09A, retcode 0.

2012-09-25 12:00:59 [INFO]: keymirror del start ----------------

2012-09-25 12:00:59 [INFO]: keymirror del for gw 4, tn 8, selfSPI 8605AB62, retcode 0.

2012-09-25 12:00:59 [PROTO_NOTIFY]: ====> IPSEC KEY DELETED <====

====> Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8605AB62/0xFFFD4D91 <====

2012-09-25 12:00:59 [INFO]: SADB_DELETE ul_proto=0 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP spi=0x8605AB62

2012-09-25 12:00:59 [INFO]: received PFKEY_DELETE seq=0 satype=ESP spi=0x8605AB62

Re: IPSec VPN restarts very often

agrgic — Sun, 25 Nov 2012 17:25:01 GMT

I had similar problem with 4.1.6, after we changed our public ip where IPSEC terminated, after that VPN didnt work. Afcourse we adjusted configs , only thing that was left is to do a restart. So i did restart dataplane, and everything start working.

Hope it helps,

Re: IPSec VPN restarts very often

rzg62s_BPC — Thu, 28 Mar 2013 04:41:32 GMT

Did you find a solution to your problem? We are experiencing the same thing on 5.0.3

Re: IPSec VPN restarts very often

chmielniak — Thu, 04 Apr 2013 08:01:20 GMT

We have disabled "Tunel Monitor" in IPSec Tunnels. It works only on PA-PA Connections.

Re: IPSec VPN restarts very often

Mark1Sewell — Wed, 10 Jul 2013 14:47:35 GMT

Hi all, We have had this issue on one of our boxes, it has been present in v4.16, v4.18h3 and v5.05.

We have stripped the VPN options back to the basics and checked all settings together but still we have issues.

This is between a PA and a Cisco ASA. It takes around 1-3hrs to re-establish the tunnel once the timeouts are hit.

I have tried clearing the SA's to refresh the tunnel but makes no difference, however restarting the dataplane although not instant

does bring the tunnel back up in around five minutes.

If PA could chime in on this is would be appreciated.

Thanks

Mark

Re: IPSec VPN restarts very often

mikand — Thu, 11 Jul 2013 08:09:42 GMT

What about the Cisco ASA?

It wouldnt be the first time Cisco has a malfunction in their code...

Re: IPSec VPN restarts very often

Support_LTC — Mon, 26 Aug 2013 18:21:39 GMT

In our case it was caused by the tunnel monitor, after un-selecting the tunnel monitor the phase 2 deletes were back to normal (instead of every 4 sec)

Re: IPSec VPN restarts very often

Unibw — Fri, 30 Aug 2013 05:43:03 GMT

Also it's worth to try to deactivate the replay-protection on phase 2. This just helped in my case dealing with inter vendor VPN.

Re: IPSec VPN restarts very often

mikand — Fri, 30 Aug 2013 20:56:57 GMT

Perhaps to identify the problem but having to have replay-protection turned off for your vpn-tunnels is just bad...

Re: IPSec VPN restarts very often

jstacey — Wed, 25 Jun 2014 11:15:42 GMT

@Support_LTC:-

This was also our experience. Your suggestion to remove the tunnel monitor resolved our identical problem. Many thanks!

Re: IPSec VPN restarts very often

infotech — Wed, 25 Jun 2014 14:12:15 GMT

why did deactivation the replay-protection resolve your issue?

Re: IPSec VPN restarts very often

Unibw — Wed, 17 Sep 2014 11:26:36 GMT

Sorry, I was not logged in for a while.

I am not really sure. The tunnel between PA-VM and ScreenOS 6.3. did not become stable for long. (300Mbit max throughput , 20ms latency, no measurable packet loss).

Re: IPSec VPN restarts very often

ahmadzubair654 — Thu, 21 Jan 2021 03:55:58 GMT

Hello,

We were also running into same issue, with NO tunnel monitors.

Every 3 seconds or 5 seconds our SPI will change, or reset to different; indicating that new 'interesting traffic' has been selected.

It was very weird behavior, certain hosts could ping fine but others wouldn't, tunnel kept resetting every 3-5 seconds.

The remote end/peer was Fortinet firewall. Turns out, our peer was doing 'strict phase 2 IP selection' in Route-Based Tunnel. In other words, in palo alto, even in route-based tunnel, we had to define proxy ID, and everything started to come normal!!!!!