Site-2-Site IPSEC Tunnel won't come online

[EDSAadmin]

i have three offices:

office 1: US - northeast    1.1.1.1  PAN-500HA

Office 2: US - southeast   2.2.2.2 PAN-3020HA  HQ site

Office 3: Shanghai China  3.3.3.3 PAN-200

all three IPSEC tunnels were up and running.  My Office 3 moved locations and when they did that we obtained a new static IP from the executive office we moved into. We updated the firewalls with the new external interface IP, IPSEC tunnel info on the local and updated all peer sites. committed the changes. after the commit office 1 and office 3 tunnel came up no problem. but office 2 and office 3 will not come up. We have deleted the config on both office 2 and office 3  and reconfigured. we have reboot both firewalls, reboot the router in office 2, we have changed the preshared key on both sites. triple checked our routes are correct in each virtual router. Contacted both ISP to make sure they are not blocking any UDP 500, 4500 or ESP traffic. they both say all IP's ports and protocols are open.

What we are seeing in the logs is office 3 is initiating ike phase 1 traffic out, but the peer box is not seeing any traffic coming in from office 3 nor is it initiating anything out to it either.  In the ike gateway for each we have it currently set to:

Exchange mode: main

IKE Crypto Profile: set at default for troubleshooting purposes

Unchecked "Enable passive mode"

Checked "Enable NAT traversal"

I have an open ticket with support but they have been unable to figure  out the problem yet. It has been escalated but i am still waiting for a callback today to continue working on it.

Has anyone run into this and if so how did you get it working again?

Any help would be greatly appreciated.

HQ Site

Shanghai Site

[ssharma]

Hi EDSAadmin,

Could you please run following on both Office2 and Office3 device :

1. Office 2 : show session all filter source 2.2.2.2 destination 3.3.3.3

                    show session all filter source 3.3.3.3 destination 2.2.2.2

2. Office 3 : show session all filter source 2.2.2.2 destination 3.3.3.3

                    show session all filter source 3.3.3.3 destination 2.2.2.2

Also please run following on both devices :

Office 2

test vpn ike-sa gateway <office3_gateway>

test vpn ipsec-sa tunnel <office3_tunnel>

show vpn ike-sa gateway <office3_gateway>

show vpn ipsec-sa tunnel <office3_tunnel>

Office 3

test vpn ike-sa gateway <office2_gateway>

test vpn ipsec-sa tunnel <office2_tunnel>

show vpn ike-sa gateway <office3_gateway>

show vpn ipsec-sa tunnel <office3_tunnel>

Also send the snapshot of system logs Monitor -> System from both devices. Thank you

[HULK]

Hello EDSAadmin,

Could you please verify on both end firewalls, that no session on the discard state. Please share below mentioned commands output. 

> show session all filter state discard

> show session all filter source-port 500 destination-port 500

> show session all filter source-port 4500 destination-port 4500

Apply Test VPN command and immidiately verify the session information as mentioned below:

test vpn ike-sa gateway <office3_gateway>

test vpn ipsec-sa tunnel <office3_tunnel>

>show session all filter source <External-IF-IP> destination <Remote-Gateway-IP>

Thanks

[EDSAadmin]

Thanks guys for the quick response. Give me a few minutes to gather that information and post it back.

[HULK]

Hello EDSAadmin,

Step-1: Could you please specify Local/peer identification ( IP) on both side firewalls.

Ste-2: apply test VPN command from CLI.

Step-3: Open an another CLI window and run > tail follow yes mp-log ikemgr.log

Please attach a screenshot of the SYSTEM logs(subtype vpn) and ikemgr.logs from the PAN firewall .

Thanks

[EDSAadmin]

HQ site:

Shanghai

HQ

Shanghai

[ssharma]

Hi EDSAadmin,

On your 1st snapshot from HQ, I would expect to see session from External to External. Instead it is showing internal to external. Could you please check your routing to verify if that is expected. Should 67.151.x.x belong to Internal zone. Thank you.

[EDSAadmin]

I exported the system logs, but i don't see a way to attach them to my posts.  Am i just blind or is that not an option on this community?

[EDSAadmin]

Interface 1/2 is in my external  zone which is 67.151.xxx.xxx

[ssharma]

Thank you for those output. I think we need to address why 67.151.x.x is showing up as internal zone. Could you please send us the output of show routing route. Thank you.

[EDSAadmin]

HQ Site

show session all filter state discard

Shanghai

[EDSAadmin]

We have multiple ISP's for redundancy. With that we use PBF rule so our default route is our backup circuit the 50.58. and then we use the PBF to send all traffic down our primary 67.151....... 

Not sure if that would be causing it to show up as an internal zone.

[EDSAadmin]

Step-1: Could you please specify Local/peer identification ( IP) on both side firewalls.  - see previous screenshots

Ste-2: apply test VPN command from CLI. did that

Step-3: Open an another CLI window and run > tail follow yes mp-log ikemgr.log

I ran the tail command and there is a lot of output.  Do you want a screenshot of this, or some other method.

Please attach a screenshot of the SYSTEM logs(subtype vpn) and ikemgr.logs from the PAN firewall .

HQ Site

Shanghai System Monitor

[HULK]

Could you please update your PAN support case number here.

Thanks

[EDSAadmin]

Here is the case number: 00252881

i added this discussion thread to the case notes as well.

Anyone got any ideas based on the information posted so far? Please let me know if you need more screenshots, logs, etc. i still owe you the tail output hulk.