IPsec Tunnel Down!

[soorajpmenon5]

Hi Team,

I'm a newbie at the Palo Alto firewall, and I've been checking the IPsec connection between PA850 at my sites. I'm encountering issues with the IPsec tunnel, which is not coming up. I tried establishing IPsec using the IP used for BGP peering, and it established without any problems. However, the problem arises when I use my own public IP configured on interface ae1.150. My public IPs are accessible on the internet and between the sites.

I've attached a screenshot showing the rules I used and the interface configuration for site1. The way I configured the interface and security rules is the same as in site2.

Can you identify any issues with my configuration? I believe that interzone communication will be okay with the way I configured the policies. Any suggestions will be highly appreciated.

Thanks!

[laurence64]

Hi

Have you checked the system logs? if you use the filter ( subtype eq 'vpn' ) you should see the logs associated with the VPN's they are usually quite good at identifying the cause.

[soorajpmenon5]

Hello Laurence,

Thanks for the suggestion, please find the logs below, IKE versions are the same on both ends.

2024/08/04 12:09:45 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:778bc0793434d002:0000000000000000.
2024/08/04 12:09:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:10:14 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:10:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:11:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:12:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:13:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:14:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:15:54 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:16:57 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:17:00 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 12:17:00 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:778bc0793434d002:0000000000000000.
2024/08/04 12:17:01 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:17:03 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:c2d454bd20682a71:0000000000000000.
2024/08/04 12:17:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:17:31 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:18:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:18:39 info vpn ike-con 0 IKE daemon configuration load phase-1 succeeded.
2024/08/04 12:19:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:19:31 info vpn ike-con 0 IKE daemon configuration load phase-2 succeeded.
2024/08/04 12:20:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:21:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:22:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:23:11 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:24:07 info vpn ike-con 0 IKE daemon configuration load phase-1 succeeded.
2024/08/04 12:24:14 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 12:24:19 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 12:24:19 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:c2d454bd20682a71:0000000000000000.
2024/08/04 12:24:19 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 14:50:23 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:1e96ac2510547e8c:0000000000000000.
2024/08/04 14:50:32 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 14:50:52 info vpn ike-gen 0 unknown ikev2 peer
2024/08/04 14:57:39 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 14:57:39 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:1e96ac2510547e8c:0000000000000000.
2024/08/04 14:57:42 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:330071f82f7dfbee:0000000000000000.
2024/08/04 15:04:55 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:04:55 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:330071f82f7dfbee:0000000000000000.
2024/08/04 15:04:59 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:4797b405643e9da3:0000000000000000.
2024/08/04 15:12:15 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:12:15 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:4797b405643e9da3:0000000000000000.
2024/08/04 15:12:17 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:89e5c77c49e05986:0000000000000000.
2024/08/04 15:19:32 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:19:32 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:89e5c77c49e05986:0000000000000000.
2024/08/04 15:19:36 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:34281788f71357b3:0000000000000000.
2024/08/04 15:26:51 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:26:51 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:34281788f71357b3:0000000000000000.
2024/08/04 15:26:54 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500]SPI:b6ada6393bea188d:0000000000000000.
2024/08/04 15:34:09 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:34:09 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:b6ada6393bea188d:0000000000000000.
2024/08/04 15:34:12 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:30eccc21cac7912f:0000000000000000.
2024/08/04 15:41:27 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:41:27 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:30eccc21cac7912f:0000000000000000.
2024/08/04 15:41:30 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500]SPI:443e5608e54c46da:0000000000000000.
2024/08/04 15:48:45 info vpn ike-gen 0 retransmission count exceeded the limit
2024/08/04 15:48:45 info vpn S2SVPN ikev2-n 0 Deleting a possible stale IKEv2 child SA. SPI:443e5608e54c46da:0000000000000000.
2024/08/04 15:48:47 info vpn S2SVPN ikev2-n 0 IKEv2 IKE SA negotiation is started as initiator, non-rekey. Initiated SA: 102.218.33.3[500]-102.218.31.3[500] SPI:d67aa6842feaf357:0000000000000000.
2024/08/04 15:49:38 info vpn ike-con 0 IKE daemon configuration load phase-1 succeeded.
2024/08/04 15:50:23 info vpn S2SVPN ikev2-s 0 IKEv2 IPSec SA delete message sent to peer. Protocol:ESP, SPI:0x9694DFA3.
2024/08/04 15:50:23 info vpn S2SVPN ipsec-k 0 IPSec key deleted. Deleted SA: 102.218.33.3[500]-102.218.31.3[500] SPI:0x00000000/0x00000000.

Thanks!

[zGomez]

You can follow the below article on troubleshooting ipsec vpn issues.

How to Troubleshoot IPSec VPN connectivity issues - Knowledge Base - Palo Alto Networks

This should help you identify why the vpn is not beeing established

[zGomez]

Are you sure you are not dropping any traffic on you firewall.  I am seeing retransmission count exceeded limit.
This could indicate you are dropping ike,ipsec esp traffic?

Can you take packet capture on the firewall or use global counters to identify if this is the case.

You can also change you settings to IKEv2 prefered then it will fall back to IKEv1 is needed.

[laurence64]

Hi,

Sorry it has taken a while to reply it does look as though there is a connectivity issue between peers or no response, if you turn up the logging level you may see more, obviously only do this if you accept that there may be a performance hit, you can use the command >debug ike tunnel <tunnel name> on debug or for the debug level of the gateway just replace tunnel <tunnel name> with gateway keyword and <gateway name>

I would also suggest using the following >tail follow yes lines 30 mp-log ikemgr.log while the connection is establishing to see what is a happening in real time,

Once you have finished return the two debug levels to normal using >debug ike tunnel <tunnel name> on normal and doing the same for the gateway to clear up.

[Edsnow]

Verify the peer to peer reachablity and verify there is a session established for port 500. for the peer and local device public ip. 

Also verify is there any security policy has been blocked the phase 1 traffic. 

[soorajpmenon5]

Thanks zGomez! - Will check that out!

[soorajpmenon5]

Hello @Edsnow @laurence64 @zGomez ,

I was out but have resumed troubleshooting the IPsec issue. Connectivity appears to be fine, and I can see the firewall is initiating an IKEv2 connection between x.x.x.x[500] and x.x.x.x[500]. I had a NAT rule that translated the subnet to the ISP IP, which I removed, but the ISP IP is still showing up. It seems that this ISP IP is causing the unknown peer error and contributing to the problem.

The warning indicating that the peer at IP address 160.212.22.202(160. is the isp IP, the ipsec local and peer ip should be my IP ends with the octect .3 on both site) is not recognized by the both side firewall on both ends causing unknown peer error.

2024-08-22 12:20:31.546 -0700 [PWRN]: x.x.x.3[500] - 160.212.22.202[45598]:0x1c007b50 unknown ikev2 peer
2024-08-22 12:22:31.548 -0700 [PERR]: Couldn't find configuration for IKE phase-1 request for peer IP 160.212.22.202[45598], ID ipaddr:x.x.x.3.

 As i have removed the SNAT, why its still taking the ISP IP, any help will be high appreciable!. I have attached the logs from ike manager.

Regards,

punkn

[Edsnow]

Hi Sooraj,

The phase 1 will initiate connection between your public IP in both end. Once the phase one completes then the phase 2 will initiate. How is your public IPs configured in the firewalls in both end. Both end ISPs are terminated in firewall or in any ISP router infront of your firewall. Also verify is there any local and remove identification has been configured. If yes  verify that's correct in both side. 

[laurence64]

Hi

It is a difficult one this, without being in front of it, But this does look like the firewall is not seeing the correct ID for the IKE to establish, check and make sure that under the IKE Profile you have the correct ID set for both local and remote peers to it must match the IP (in this case) of the opposing firewall, in the last log you sent it does say that it cannot find the configuration for the peer ID, so there is an issue with that local/remote configuration, or it could be the Local / Peer IP addresses in IKE config, but that really does look like where the problem is.

[soorajpmenon5]

Hi @laurence64 @Edsnow ,

Both IPs are terminated on the firewall's aggregated interface, ae1.150, within a custom zone named untrust-internal. The ISP's physical connectivity is established through a switch, and BGP peering is configured directly on the firewall. Not sure why peer ID appearing on the other end firewall as the ISP's IP address and local and peer IPs are configured correctly on both sides.

FYI - The IKE gateway I'm using for this tunnel was previously configured for the ISP IP to establish the tunnel. I'm simply changing the local and remote IPs to my public IP by selecting the corresponding interface. I believe this should be fine, and there should be no need to create a new IKE gateway.

Thanks!

[laurence64]

Hi,

So, if both IP's are reachable (as in they are routable and they are not in zones that are separate from the outside zone thus needing rules) and the local and remote IP's are configured correctly in the WebUI then there is something else wrong that we cannot see, if I am right the IP's that are in the logs are not the correct IP's? but are the old ones, what you may have is some stuck config, this can happen from time to time and although from the WebUI all looks correct there could be a stale reference to the previous configuration in the XML,

I would go through the command line, change the output to set and see if you get a result when matching on the old IP, I agree that in general it is reasonable to assume that you would not have to create a new IKE gateway when changing IP's however it is something that I would do, ultimately the troubleshooting is returning very odd results when you compare it to the configuration so starting with a clean slate may be the only alternative.

Incase you don't have them the commands are below.

> set cli config-output-format set

configure

# show | match 'Your variable