IPSec VPN restarts very often

[chmielniak]

Hallo,

I have defined a IPSec VPN connection with following params:

ike: 3des/sha1/dh5 Lifetime: 8 hours

ipsec: ESP/3des/sha1/dh5 Lifetime: 30 minutes (life size not set, shows 0MB)

ike gateway: main mode, DP enabled

The connection is established but in system log I see very often (every 5 sec.) tunnel is again and again down and up. We have packet lost about 0.5%.

Any ideas? I've already configured the connection from scratch again.

Jacek.

Log file:

2012/09/24 12:36:39    ipsec-key-delete    IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8C5FC8B5/0xFFFD0AD9.

2012/09/24 12:36:39    ike-send-p2-delete    IKE protocol IPSec SA delete message sent to peer. SPI:0x8C5FC8B5.

2012/09/24 12:36:38    ipsec-key-install    IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xDF1F9E37/0xFFFD0ADA lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:38    ike-nego-p2-succ    IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xCFE39FEB, SPI:0xDF1F9E37/0xFFFD0ADA.

2012/09/24 12:36:38    ike-nego-p2-start    IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xCFE39FEB.

2012/09/24 12:36:35    ipsec-key-delete    IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xCDCD7E83/0xFFFD0AD8.

2012/09/24 12:36:35    ike-send-p2-delete    IKE protocol IPSec SA delete message sent to peer. SPI:0xCDCD7E83.

2012/09/24 12:36:34    ipsec-key-install    IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8C5FC8B5/0xFFFD0AD9 lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:34    ike-nego-p2-succ    IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x756F7417, SPI:0x8C5FC8B5/0xFFFD0AD9.

2012/09/24 12:36:34    ike-nego-p2-start    IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x756F7417.

2012/09/24 12:36:31    ipsec-key-delete    IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xE36D50CD/0xFFFD0AD7.

2012/09/24 12:36:31    ike-send-p2-delete    IKE protocol IPSec SA delete message sent to peer. SPI:0xE36D50CD.

2012/09/24 12:36:30    ipsec-key-install    IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xCDCD7E83/0xFFFD0AD8 lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:30    ike-nego-p2-succ    IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x43C3E41C, SPI:0xCDCD7E83/0xFFFD0AD8.

2012/09/24 12:36:30    ike-nego-p2-start    IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x43C3E41C.

2012/09/24 12:36:27    ipsec-key-delete    IPSec key deleted. Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8D0BBED9/0xFFFD0AD6.

2012/09/24 12:36:27    ike-send-p2-delete    IKE protocol IPSec SA delete message sent to peer. SPI:0x8D0BBED9.

2012/09/24 12:36:26    ipsec-key-install    IPSec key installed. Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xE36D50CD/0xFFFD0AD7 lifetime 1800 Sec lifesize unlimited.

2012/09/24 12:36:26    ike-nego-p2-succ    IKE phase-2 negotiation is succeeded as initiator, quick mode. Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x15CF19C6, SPI:0xE36D50CD/0xFFFD0AD7.

2012/09/24 12:36:26    ike-nego-p2-start    IKE phase-2 negotiation is started as initiator, quick mode. Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x15CF19C6.

[mikand]

Since it takes two to tango VPN - what do you have at the other side?

And if possible, could you put one of those peers much closer to your PA to rule out any interference from the network(s) in between?

Also, how is your security policies setup for this traffic?

And is your ipsec setup on a physical interface (which perhaps goes up and down?) or a loopback interface?

[chmielniak]

I have 3 VPNs running. The problem appeared after some update in 4.1.x with all 3 connections. I can control both sides of only one connection (it's cisco router IPSec with VTI). I've just deleted the configuration on both sides and recreated it with the same parameters and it works now.

The other two VPN partners haven't change a thing and it shows above problem.

All IPSec connections are setup on the same physical interface together with normal internet traffic. I observe no problems with the interface. Security policies allow ssh connections outgoing.

How can I see what makes the connections up and down?

[ppatel]

Make sure the Crypto settings are same on both the sides and try initiating the tunnel traffic from the remote side.

Also try configuring the ipsec-crypto to DH group to "no-pfs" on both the sides. Clear the VPN tunnels on the Palo Alto side.

admin@PA> clear vpn ike-sa gateway

  <value>  clear for given IKE gateway

admin@PA> clear vpn ipsec-sa tunnel

  <value>  clear for given VPN tunnel

Try initiate the tunnel from the cisco side.

Monitor the system logs on the firewall to see the IPSEC negotiation. Check to see if the tunnel comes up.

Which version of software are you using on the firewall? There is an issue with software release 4.1.5 where after an upgrade or intermittently IPSec VPN tunnel would not come up when Palo Alto Networks firewall initiates a

connection to a Cisco ASA device. [Bug # 39884] This issue was addressed in S/w 4.1.6.

Let me know if this helps.

Regards

Parth

[chmielniak]

We have 4.1.7.

I cannot control the other side of tunnel. Those are partners, that request configuration according to their policy. I have checked the ipsec params requested. They are set correct on both sides. IPSec tunnel is established, but in log i can see those IKE deletes every ca. 10 sec. I had to disable information log on syslog.

One on VPNS ends on Fortinet. The only one VPN, that is working has no Proxy-ID defined. The other two with problems have Proxy Ids defined in IPSec tunnel.

I can see any reason for "IKE protocol IPSec SA delete message sent to peer". We experience long RTT and packet lost on those connections.

Jacek.

[ppatel]

Try to have a look at the ikemgr log on the firewall.

admin@PA> tail follow yes mp-log ikemgr.log

The above log will be able to give detailed logging on the ike negotiation process that is done by the firewall

Try to clear the vpn flow using the commands that were given before and run the tail command.

Please open a support ticket with TAC to troubleshoot the issue if it is still not resolving. It would be great to have access to both the sides of the tunnel to debug/troubleshoot Ipsec.

Regards

Parth

[chmielniak]

I have disabled one vpn, so that I could cut the debug log for only one VPN. I set level with: debug ike global on normal
I have resetted the connection. I have cut three phases delete/establish: (XX.XX.XX.XX is own IP, YY.YY.YY.YY is partner IP)

2012-09-25 12:00:47 [INFO]: panike_keyacquire_cb 2fe29fa8

2012-09-25 12:00:47 [INFO]: IPsec-SA request for YY.YY.YY.YY queued since no phase1 found

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-1 NEGOTIATION STARTED AS INITIATOR, MAIN MODE <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] cookie:6041d73ad3ab677f:0000000000000000 <====

2012-09-25 12:00:47 [INFO]: received Vendor ID: DPD

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-1 NEGOTIATION SUCCEEDED AS INITIATOR, MAIN MODE <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] cookie:6041d73ad3ab677f:90e8c84f5cf54d25 lifetime 28800 Sec <====

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION STARTED AS INITIATOR, (QUICK MODE) <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x9395643A <====

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION SUCCEEDED AS INITIATOR, (QUICK MODE) <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x9395643A, SPI:0xC1B84E56/0xFFFD4D90 <====

2012-09-25 12:00:47 [INFO]: SADB_UPDATE ul_proto=255 src=YY.YY.YY.YY[500] dst=XX.XX.XX.XX[500] satype=ESP samode=tunl spi=0xC1B84E56 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:47 [INFO]: SADB_ADD ul_proto=255 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP samode=tunl spi=0xFFFD4D90 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:47 [INFO]: IPsec-SA established: ESP/Tunnel YY.YY.YY.YY[500]->XX.XX.XX.XX[500] spi=3250081366(0xc1b84e56)

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> IPSEC KEY INSTALLATION SUCCEEDED <====

====> Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xC1B84E56/0xFFFD4D90 lifetime 1800 Sec lifesize unlimited <====

2012-09-25 12:00:47 [INFO]: keymirror add start ++++++++++++++++

2012-09-25 12:00:47 [INFO]: keymirror add for gw 4, tn 8, selfSPI C1B84E56, retcode 0.

2012-09-25 12:00:47 [INFO]: keymirror del start ----------------

2012-09-25 12:00:47 [INFO]: keymirror del for gw 4, tn 8, selfSPI F703F0F5, retcode 0.

2012-09-25 12:00:47 [PROTO_NOTIFY]: ====> IPSEC KEY DELETED <====

====> Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xF703F0F5/0xFFFD4D8E <====

2012-09-25 12:00:47 [INFO]: SADB_DELETE ul_proto=0 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP spi=0xF703F0F5

2012-09-25 12:00:47 [INFO]: received PFKEY_DELETE seq=0 satype=ESP spi=0xF703F0F5

2012-09-25 12:00:54 [INFO]: panike_keyacquire_cb 2fe2ad98

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION STARTED AS INITIATOR, (QUICK MODE) <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x173FBEFA <====

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION SUCCEEDED AS INITIATOR, (QUICK MODE) <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0x173FBEFA, SPI:0x8605AB62/0xFFFD4D91 <====

2012-09-25 12:00:54 [INFO]: SADB_UPDATE ul_proto=255 src=YY.YY.YY.YY[500] dst=XX.XX.XX.XX[500] satype=ESP samode=tunl spi=0x8605AB62 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:54 [INFO]: SADB_ADD ul_proto=255 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP samode=tunl spi=0xFFFD4D91 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:54 [INFO]: IPsec-SA established: ESP/Tunnel YY.YY.YY.YY[500]->XX.XX.XX.XX[500] spi=2248518498(0x8605ab62)

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> IPSEC KEY INSTALLATION SUCCEEDED <====

====> Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8605AB62/0xFFFD4D91 lifetime 1800 Sec lifesize unlimited <====

2012-09-25 12:00:54 [INFO]: keymirror add start ++++++++++++++++

2012-09-25 12:00:54 [INFO]: keymirror add for gw 4, tn 8, selfSPI 8605AB62, retcode 0.

2012-09-25 12:00:54 [INFO]: keymirror del start ----------------

2012-09-25 12:00:54 [INFO]: keymirror del for gw 4, tn 8, selfSPI C1B84E56, retcode 0.

2012-09-25 12:00:54 [PROTO_NOTIFY]: ====> IPSEC KEY DELETED <====

====> Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0xC1B84E56/0xFFFD4D90 <====

2012-09-25 12:00:54 [INFO]: SADB_DELETE ul_proto=0 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP spi=0xC1B84E56

2012-09-25 12:00:54 [INFO]: received PFKEY_DELETE seq=0 satype=ESP spi=0xC1B84E56

2012-09-25 12:00:58 [INFO]: panike_keyacquire_cb 2fe29fa8

2012-09-25 12:00:58 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION STARTED AS INITIATOR, (QUICK MODE) <====

====> Initiated SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xFB43C451 <====

2012-09-25 12:00:58 [PROTO_NOTIFY]: ====> PHASE-2 NEGOTIATION SUCCEEDED AS INITIATOR, (QUICK MODE) <====

====> Established SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] message id:0xFB43C451, SPI:0x9838E09A/0xFFFD4D92 <====

2012-09-25 12:00:58 [INFO]: SADB_UPDATE ul_proto=255 src=YY.YY.YY.YY[500] dst=XX.XX.XX.XX[500] satype=ESP samode=tunl spi=0x9838E09A authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:58 [INFO]: SADB_ADD ul_proto=255 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP samode=tunl spi=0xFFFD4D92 authtype=SHA1 enctype=3DES lifetime soft time=1800 bytes=0 hard time=1800 bytes=0

2012-09-25 12:00:58 [INFO]: IPsec-SA established: ESP/Tunnel YY.YY.YY.YY[500]->XX.XX.XX.XX[500] spi=2553864346(0x9838e09a)

2012-09-25 12:00:58 [PROTO_NOTIFY]: ====> IPSEC KEY INSTALLATION SUCCEEDED <====

====> Installed SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x9838E09A/0xFFFD4D92 lifetime 1800 Sec lifesize unlimited <====

2012-09-25 12:00:58 [INFO]: keymirror add start ++++++++++++++++

2012-09-25 12:00:58 [INFO]: keymirror add for gw 4, tn 8, selfSPI 9838E09A, retcode 0.

2012-09-25 12:00:59 [INFO]: keymirror del start ----------------

2012-09-25 12:00:59 [INFO]: keymirror del for gw 4, tn 8, selfSPI 8605AB62, retcode 0.

2012-09-25 12:00:59 [PROTO_NOTIFY]: ====> IPSEC KEY DELETED <====

====> Deleted SA: XX.XX.XX.XX[500]-YY.YY.YY.YY[500] SPI:0x8605AB62/0xFFFD4D91 <====

2012-09-25 12:00:59 [INFO]: SADB_DELETE ul_proto=0 src=XX.XX.XX.XX[500] dst=YY.YY.YY.YY[500] satype=ESP spi=0x8605AB62

2012-09-25 12:00:59 [INFO]: received PFKEY_DELETE seq=0 satype=ESP spi=0x8605AB62

[agrgic]

I had similar problem with 4.1.6, after we changed our public ip where IPSEC terminated, after that VPN didnt work. Afcourse we adjusted configs , only thing that was left is to do a restart. So i did restart dataplane, and everything start working.

Hope it helps,

[rzg62s_BPC]

Did you find a solution to your problem? We are experiencing the same thing on 5.0.3

[chmielniak]

We have disabled "Tunel Monitor" in IPSec Tunnels. It works only on PA-PA Connections.

[Mark1Sewell]

Hi all, We have had this issue on one of our boxes, it has been present in v4.16, v4.18h3 and v5.05.

We have stripped the VPN options back to the basics and checked all settings together but still we have issues.

This is between a PA and a Cisco ASA. It takes around 1-3hrs to re-establish the tunnel once the timeouts are hit.

I have tried clearing the SA's to refresh the tunnel but makes no difference, however restarting the dataplane although not instant

does bring the tunnel back up in around five minutes.

If PA could chime in on this is would be appreciated.

Thanks

Mark

[mikand]

What about the Cisco ASA?

It wouldnt be the first time Cisco has a malfunction in their code...

[Support_LTC · Accepted solution]

In our case it was caused by the tunnel monitor, after un-selecting the tunnel monitor the phase 2 deletes were back to normal (instead of every 4 sec)

[Unibw]

Also it's worth to try to deactivate the replay-protection on phase 2. This just helped in my case dealing with inter vendor VPN.

[mikand]

Perhaps to identify the problem but having to have replay-protection turned off for your vpn-tunnels is just bad...