topic Re: Site-2-Site IPSEC Tunnel won't come online in General Topics

Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 14:15:15 GMT

i have three offices:

office 1: US - northeast 1.1.1.1 PAN-500HA

Office 2: US - southeast 2.2.2.2 PAN-3020HA HQ site

Office 3: Shanghai China 3.3.3.3 PAN-200

all three IPSEC tunnels were up and running. My Office 3 moved locations and when they did that we obtained a new static IP from the executive office we moved into. We updated the firewalls with the new external interface IP, IPSEC tunnel info on the local and updated all peer sites. committed the changes. after the commit office 1 and office 3 tunnel came up no problem. but office 2 and office 3 will not come up. We have deleted the config on both office 2 and office 3 and reconfigured. we have reboot both firewalls, reboot the router in office 2, we have changed the preshared key on both sites. triple checked our routes are correct in each virtual router. Contacted both ISP to make sure they are not blocking any UDP 500, 4500 or ESP traffic. they both say all IP's ports and protocols are open.

What we are seeing in the logs is office 3 is initiating ike phase 1 traffic out, but the peer box is not seeing any traffic coming in from office 3 nor is it initiating anything out to it either. In the ike gateway for each we have it currently set to:

Exchange mode: main

IKE Crypto Profile: set at default for troubleshooting purposes

Unchecked "Enable passive mode"

Checked "Enable NAT traversal"

I have an open ticket with support but they have been unable to figure out the problem yet. It has been escalated but i am still waiting for a callback today to continue working on it.

Has anyone run into this and if so how did you get it working again?

Any help would be greatly appreciated.

HQ Site

Shanghai Site

Re: Site-2-Site IPSEC Tunnel won't come online

ssharma — Wed, 24 Sep 2014 14:27:50 GMT

Hi EDSAadmin,

Could you please run following on both Office2 and Office3 device :

1. Office 2 : show session all filter source 2.2.2.2 destination 3.3.3.3

show session all filter source 3.3.3.3 destination 2.2.2.2

2. Office 3 : show session all filter source 2.2.2.2 destination 3.3.3.3

show session all filter source 3.3.3.3 destination 2.2.2.2

Also please run following on both devices :

Office 2

test vpn ike-sa gateway <office3_gateway>

test vpn ipsec-sa tunnel <office3_tunnel>

show vpn ike-sa gateway <office3_gateway>

show vpn ipsec-sa tunnel <office3_tunnel>

Office 3

test vpn ike-sa gateway <office2_gateway>

test vpn ipsec-sa tunnel <office2_tunnel>

show vpn ike-sa gateway <office3_gateway>

show vpn ipsec-sa tunnel <office3_tunnel>

Also send the snapshot of system logs Monitor -> System from both devices. Thank you

Re: Site-2-Site IPSEC Tunnel won't come online

HULK — Wed, 24 Sep 2014 14:37:05 GMT

Hello EDSAadmin,

Could you please verify on both end firewalls, that no session on the discard state. Please share below mentioned commands output.

> show session all filter state discard

> show session all filter source-port 500 destination-port 500

> show session all filter source-port 4500 destination-port 4500

Apply Test VPN command and immidiately verify the session information as mentioned below:

test vpn ike-sa gateway <office3_gateway>

test vpn ipsec-sa tunnel <office3_tunnel>

>show session all filter source <External-IF-IP> destination <Remote-Gateway-IP>

Thanks

Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 14:38:59 GMT

Thanks guys for the quick response. Give me a few minutes to gather that information and post it back.

Re: Site-2-Site IPSEC Tunnel won't come online

HULK — Wed, 24 Sep 2014 14:43:22 GMT

Hello EDSAadmin,

Step-1: Could you please specify Local/peer identification ( IP) on both side firewalls.

Ste-2: apply test VPN command from CLI.

Step-3: Open an another CLI window and run > tail follow yes mp-log ikemgr.log

Please attach a screenshot of the SYSTEM logs(subtype vpn) and ikemgr.logs from the PAN firewall .

Thanks

Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 15:10:35 GMT

HQ site:

Shanghai

Re: Site-2-Site IPSEC Tunnel won't come online

ssharma — Wed, 24 Sep 2014 15:17:07 GMT

Hi EDSAadmin,

On your 1st snapshot from HQ, I would expect to see session from External to External. Instead it is showing internal to external. Could you please check your routing to verify if that is expected. Should 67.151.x.x belong to Internal zone. Thank you.

Re: Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 15:20:34 GMT

I exported the system logs, but i don't see a way to attach them to my posts. Am i just blind or is that not an option on this community?

Re: Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 15:23:46 GMT

Interface 1/2 is in my external zone which is 67.151.xxx.xxx

Re: Site-2-Site IPSEC Tunnel won't come online

ssharma — Wed, 24 Sep 2014 15:42:23 GMT

Thank you for those output. I think we need to address why 67.151.x.x is showing up as internal zone. Could you please send us the output of show routing route. Thank you.

Re: Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 15:44:10 GMT

HQ Site

show session all filter state discard

Shanghai

Re: Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 15:50:35 GMT

We have multiple ISP's for redundancy. With that we use PBF rule so our default route is our backup circuit the 50.58. and then we use the PBF to send all traffic down our primary 67.151.......

Not sure if that would be causing it to show up as an internal zone.

Re: Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 16:05:30 GMT

Step-1: Could you please specify Local/peer identification ( IP) on both side firewalls. - see previous screenshots

Ste-2: apply test VPN command from CLI. did that

Step-3: Open an another CLI window and run > tail follow yes mp-log ikemgr.log

I ran the tail command and there is a lot of output. Do you want a screenshot of this, or some other method.

Please attach a screenshot of the SYSTEM logs(subtype vpn) and ikemgr.logs from the PAN firewall .

HQ Site

Shanghai System Monitor

Re: Re: Site-2-Site IPSEC Tunnel won't come online

HULK — Wed, 24 Sep 2014 19:47:33 GMT

Could you please update your PAN support case number here.

Thanks

Re: Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Wed, 24 Sep 2014 21:33:19 GMT

Here is the case number: 00252881

i added this discussion thread to the case notes as well.

Anyone got any ideas based on the information posted so far? Please let me know if you need more screenshots, logs, etc. i still owe you the tail output hulk.

Re: Re: Site-2-Site IPSEC Tunnel won't come online

HULK — Thu, 25 Sep 2014 22:02:04 GMT

Good to see, the case has been closed today.

Thanks

Re: Site-2-Site IPSEC Tunnel won't come online

EDSAadmin — Fri, 26 Sep 2014 13:25:22 GMT

We found the problem. On our HQ firewall we have a clean up rule set. When we put that in initially it caused all of our tunnels to go down since it was dropping the ike phase 1 traffic. We put a VPN tunnel policy in place, with source zone external and all four site external IP's and destination zone with all four external IP's one for each site allowing any app, and any service. Stupid me i forgot we had to set that up and that policy had the old IP address in it. once we updated the policy with the new IP and commited the tunnel came up no problem.

Thanks for all your help.

Re: Site-2-Site IPSEC Tunnel won't come online

Retired Member — Fri, 26 Sep 2014 15:36:24 GMT

good to hear that is solved.implicity deny should be written always carefully.