IPSec Tunnel fails after 1 packet

[Bradmatix]

Hi Guys,

We have a number of Palo Alto firewalls at our satellite sites configured in a Mesh VPN.

Site A, Site B, and Site C (Internal) all work successfully.

Site C DMZ can establish a tunnel to all the other sites, however as soon as the VPN is used, it immediately stops working.

Site C Internal and Site C DMZ are different Virtual Routers running on the same vsys.

Testing the VPN from SiteA to DMZ works

admin@SiteA(active)> test vpn ike-sa gateway DMZ

Start time: Jun.30 16:48:21
Initiate 1 IKE SA.

Sending a ping across the tunnel works for a single packet.
admin@SiteA(active)> ping source 10.1.1.1 host 100.1.1.1
PING 100.1.1.1 (100.1.1.1) from 10.1.1.1 : 56(84) bytes of data.
64 bytes from 100.1.1.1: icmp_seq=1 ttl=64 time=2.09 ms
^C
--- 100.1.1.1 ping statistics ---
5 packets transmitted, 1 received, 80% packet loss, time 4058ms
rtt min/avg/max/mdev = 2.099/2.099/2.099/0.000 ms

Testing another separate ping fails
admin@SiteA(active)> ping source 10.1.1.1 host 100.1.1.1
PING 100.1.1.1 (100.1.1.1) from 10.1.1.1 : 56(84) bytes of data.
^C
--- 100.1.1.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1018ms

Using TEST VPN to reinitiate some keys

admin@SiteA(active)> test vpn ike-sa gateway DMZ

Start time: Jun.30 16:48:51
Initiate 1 IKE SA.

Ping works for 1 packet.

admin@SiteA(active)> ping source 10.1.1.1 host 100.1.1.1
PING 100.1.1.1 (100.1.1.1) from 10.1.1.1 : 56(84) bytes of data.
64 bytes from 100.1.1.1: icmp_seq=1 ttl=64 time=2.31 ms
^C
--- 100.1.1.1 ping statistics ---
5 packets transmitted, 1 received, 80% packet loss, time 4082ms
rtt min/avg/max/mdev = 2.310/2.310/2.310/0.000 ms

We are seeing the exact same behaviour from 

Site A -> Site C DMZ

Site B -> Site C DMZ

Site C Internal -> Site C DMZ

Some additional info:

• This Mesh VPN was working prior to a cutover to Palo Alto firewalls. No Rules/Routing was changed, only the firewall devices. No blocks being seen on any devices between these firewalls.
• DH groups, keys, etc have been checked. The VPN establishes successfully.
• There is no Maximum Lifesize set on the tunnels.
• Checked all the static routes. Traffic is going through the correct tunnels. Traffic in the logs looks correct
• Have tried to change IPSec crypto suites

Has anyone seen similar behaviour before or have any suggestions on what we can check next? Only thing I can think is that the Palo Alto doesn't like the 2 VPNs being created on the same vsys, despite different Virtual Routers, but would have thought other people would be doing this?

[OtakarKlier]

Hello,

How is routing handled, is it static or a dynamic protocol?

Regards,

[BPry]

@Bradmatix,

Additionally to what @OtakarKlier mentioned, are we dealing with only PAN firewalls here? SiteA is a PAN clearly, but what firewall vendor are we working with on SiteB or SiteC? 

[Bradmatix]

@OtakarKlier - All the tunnel routing has been added statically. We also have a default route so traffic not destined for the tunnel can go out.

@BPry - All 3 are PAN firewalls

Site A - PA 3220

Site B - PA 850

Site C - PA 3250