Active/Active & IPSec Trouble

dshue — Mon, 07 Apr 2014 20:56:10 GMT

After implementing HA Active/Active, we left S2S VPN tunnels alone. Ultimately no changes to IKE Gateways. The S2S terminate to a /30 address that is statically routed from the ISP to ethernet1/12 on the active-primary. Tunnel interfaces and their routes are identical on both primary and secondary, IKE Gateways are NOT synced as this is just an IP assigned to the 1/12 interface on the primary side. OSPF on the internal LAN side, HOWEVER, supplied static routes for the remote sites to only route to primary device. Secondary should never be session owner.

Session Owner = First Packet

Session Setup = ip-modulo

Phase 1 and Phase 2 seem to come up fine. Communication is established and devices can talk fine. However, the key lifetime is set for 2 hours. Throughout the day, the remote side will stop responding at key expiration. It appears that a new negotiation is started and completed, however for the two hour key time the VPN tunnels will not pass traffic. This doesn't happen every time it negotiates, and I can seem to locate a pattern. The best lead that I have right now is that the return traffic from the remote side is being discarded due to the flow being in a discard state. I can also see where the 'decap packets' in the vpn flow will not increment. I've looked through the ike debug log and don't really see anything going on there during this state of limbo. Phase 1 appears to be up, phase 2 SAs seem to be up, but for some reason the firewall is rejecting these return packets and I cannot determine why. If can clear phase 1, and the flows and it will pick right back up. Other option is to just wait for Phase 1 to timeout and re-key. These devices were working fine prior to HA to the best of my knowledge.

Any help would be greatly appreciated!

== Apr 07 09:33:26 ==

Packet received at ha_ingress stage

Packet info: len 1532 port 49 interface 49 vsys 1

wqe index 228383 packet 0x0x8000000416f718e0

Packet decoded dump:

L2: 00:1b:17:3c:21:31->00:1b:17:c8:6f:31, type 0x7261

L3 binary dump: 40 bytes

00000000: 80 02 00 1b 00 1b 17 3c 21 1b 00 16 4d 6e a6 8c .......< !...Mn..

00000010: 08 00 45 00 05 dc c8 d9 40 00 36 06 ff 43 c7 1b ..E..... @.6..C..

00000020: 4e b8 41 29 20 02 00 50 N.A) ..P

...skipping...

IP: 166.xxx.xxx.xxx->71.xxx.xxx.xxx, protocol 50

version 4, ihl 5, tos 0x00, len 120,

id 663, frag_off 0x0000, ttl 49, checksum 57440

L4 binary dump: 16 bytes

00000000: f3 46 fb c0 00 00 01 3c 19 ac 55 94 f5 04 3f a5 .F.....< ..U...?.

Flow lookup, key word0 0xf346fbc000023200 word1 0

Flow 211818 found, state 3

Packet dropped, flow in discard state

Session 105909

c2s flow:

source: 166.xxx.xxx.xxx [untrust]

dst: 71.2.234.230

proto: 50

sport: 62278 dport: 64448

state: DISCARD type: FLOW

src user: unknown

dst user: unknown

s2c flow:

source: 71.xxx.xxx.xxx [untrust]

dst: 166.142.205.229

proto: 50

sport: 64448 dport: 62278

state: DISCARD type: FLOW

src user: unknown

dst user: unknown

start time : Mon Apr 7 09:28:39 2014

timeout : 60 sec

time to live : 58 sec

total byte count(c2s) : 103868

total byte count(s2c) : 0

layer7 packet count(c2s) : 722

layer7 packet count(s2c) : 0

vsys : vsys1

application : ipsec-esp

rule : ALLOW_IN_VPN

session to be logged at end : True

session in session ager : True

session synced from HA peer : False

session owned by local HA A/A : True

layer7 processing : completed

URL filtering enabled : True

URL category : any

session via syn-cookies : False

session terminated on host : True

session traverses tunnel : False

captive portal session : False

ingress interface : ethernet1/12

egress interface : ethernet1/12

session QoS rule : N/A (class 4)

session tracker stage deny : host service

session tracker stage l7proc : ctd app has no decoder

Re: Active/Active & IPSec Trouble

jdelio — Tue, 08 Apr 2014 15:56:51 GMT

What version are you at now?

I have seen a similar A/A issue with vpn's that break in the middle of the key-renegotiate with 5.0.9.

There are a couple of solutions..

1. Upgrade to 6.0.1, and this issue should go away.

2. Test with 1 member shut off/disabled. IF that works, then you can try to Configure Active/Passive, but not sure if that is an option for you.

I hope this helps.

Re: Active/Active & IPSec Trouble

dshue — Tue, 08 Apr 2014 19:19:15 GMT

Thank you jdelio, I think you nailed it!

We are at 5.0.11 and just upgraded to that a week prior to implementing HA.

I've had a support case open since starting with Active/Active. They responded today that it is most likely they bug you are talking about but I'm reluctant to upgrade to 6.0.1 however.

From the 6.0.1 Release Notes - Resolved Issues

• 60201—In a HA Active/Active setup, an IPSec key renegotiation timing issue caused the new IPSec session to be set to DISCARD until the next key renegotiation. This caused traffic loss until the next tunnel key renegotiation.

Re: Active/Active & IPSec Trouble

dshue — Tue, 08 Apr 2014 19:21:22 GMT

Have you had any issues with 6.0.1?

Re: Active/Active & IPSec Trouble

jdelio — Tue, 08 Apr 2014 21:09:09 GMT

Honestly, there have been some issues here and there, impossible not to have any.. but they were either resolved with rebooting the unit a couple of times to clear a stuck state or ensuring that there is a subnet mask on the IPSEC/VPN tunnel interfaces. Other than that, I would recommend looking through the "known issues" in the release notes.

Glad I could help.

topic Active/Active & IPSec Trouble in General Topics

Active/Active & IPSec Trouble

Re: Active/Active & IPSec Trouble

Re: Active/Active & IPSec Trouble

Re: Active/Active & IPSec Trouble

Re: Active/Active & IPSec Trouble