IPSec Dynamic Peer VPN, failure to send traffic over attached tunnel interface

[Adrian_Jensen]

Is anyone aware of a known issue with sending traffic over an IPSec tunnel interface when using multiple dynamic peers with FQDN (host) peer identification?

I have multiple existing branch locations connected to the PA with IKEv2 IPSec tunnels using dynamic FQDN (host) peer identification from Cisco branch routers. Up to now it has worked fine, no problems re-establishing traffic after outages/reboots. After adding a new IPSec yesterday for a new branch, an old branch location broke overnight (forced new IKE key after upstream router reboot, IKE/IPSec rekey worked fine up till then). The correct IPSec gateway/tunnel comes up immediately (phase 1/2 complete normally, no errors), but the attached tunnel will not pass traffic. Both sides show outbound packets but no inbound packets received over the IPSec tunnel. I believe the PaloAlto is dropping the traffic in both directions on the tunnel (or trying to send through the wrong tunnel), but nothing shows as dropped or misrouted in logging.

I was unable to get the failed site to pass traffic until I disabled the new branch IKE/IPSec. Both IKE gateways have unique FQDNs. Among all the branch locations, the only difference is that the failing location is set as Passive/NAT as it is behind a CGN.

mp_log ikemgr.log shows the initial connection as matching the new branch setup (expected as Branch_07 is the first available dynamic peer match), followed by switching to the expected Branch_22 setup and tunnel based on received FQDN and attaching to the correct tunnel:

: received IKE request xx.xx.xx.xx[37512] to xx.xx.xx.xx[500], found IKE gateway Branch_07
: ====> IKEv2 IKE SA NEGOTIATION STARTED AS RESPONDER, non-rekey; gateway Branch_07 <====
  ====> Initiated SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] SPI:xxx:zzz SN:49662 <====
...
: xx.xx.xx.xx[37512] - xx.xx.xx.xx[500]:0x555555zzz received ID_I (type fqdn [abc.def.ghi]) matches IKE gateway Branch_22
: [IKE SA hashtbl update] from 2 to 10 (new gw: Branch_22).
: xx.xx.xx.xx[500] - xx.xx.xx.xx[37512]:yyy authentication result: success
...
: ====> IKEv2 CHILD SA NEGOTIATION STARTED AS RESPONDER, non-rekey; gateway Branch_22 <====
  ====> Initiated SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] message id:0x00000001 parent SN:49662 <====
...
: ====> IPSEC KEY INSTALLATION SUCCEEDED; tunnel Branch_22 <====
====> Installed SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] SPI:yyy/zzz lifetime 3600 Sec lifesize unlimited <====
...
: ====> IKEv2 IKE SA NEGOTIATION SUCCEEDED AS RESPONDER, non-rekey; gateway Branch_22 <====
  ====> Established SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] SPI:yyy:zzz SN:49662 lifetime 28800 Sec <====

[Adrian_Jensen · Accepted solution]

After discussing with PaloAlto support and a bunch of digging, this is a known limitation of IPSec tunnels with dynamic peers, which is documented in an internal KB, but I have not been able to find it on the public KB. In short, if any IKE dynamic peer connection uses NAT Traversal, all IKE dynamic peers must be set to NAT Traversal. The PaloAlto IKE NAT Traversal option does not force NAT, but instead enables the ability to exchange initial NAT discovery information. This follow-up is to document the problem for anyone searching in the future:

When using multiple IKE Gateways with dynamic peers (no static remote end IP), a peer is identified using a FQDN (host), KEYID(hex), or FQDN(email) to match it to a particular Gateway configuration. This identification exchange occurs in the second packet exchange (3rd/4th packets) of the initial IKE TCP/500 connection.

When an IKE peer is behind NAT, it sends NAT discovery options in the initial packet exchange (1st/2nd packets) of the IKE TCP/500 connection. These are ISAKMP option 41 messages "NAT_DETECTION_SOURCE_IP" and "NAT_DETECTION_DESTINATION_IP", which contain hashes of the sender's/receiver's original IP addresses (before an intermediate NAT has changed the IP packets). The peers compare the received hashed to the received packet IP addresses to determine if NAT has taken place. If NAT is not detected, communication switches from TCP/500 to ESP (no port number). If NAT is detected, communication switches to UDP/4500 packet  encapsulation.

When a new IKE connection request is received on the PaloAlto, and it does not match a static peer address, it is matched to the first available IKE Gateway dynamic peer configuration (that is not already connected). If this Gateway configuration does not have NAT Traversal selected, the PaloAlto will not respond with NAT discovery options, even if the connecting peer has sent them. In the second packet exchange, the peer identifier is sent and the PaloAlto will re-match the connection to an IKE Gateway with corresponding peer information (or drop the connection if no matching peer configuration is found). The encryption settings will be applied and phase 1/2 security keys negotiated.

If multiple dynamic peer IKE Gateway configurations exist, an incoming peer connection requiring NAT may match a peer configuration without NAT selected (depending on which Gateways are currently available). As the NAT discovery options have already occurred, before a dynamic peer is matched to the correct configuration, no NAT discovery will occur and necessary UDP encapsulation to transit NAT will not happen. The phase 1/2 (IKE/IPSec) will establish, security keys renew, and outbound traffic is shown in the encrypted tunnel, but no traffic will actually be received at the peer.

If any IKE Gateway dynamic peer has NAT Traversal selected, all dynamic peer configurations must have NAT Traversal selected to ensure NAT discovery will occur on the PaloAlto. If NAT Traversal is selected on an IKE Gateway configuration and the matching peer does not require NAT (as determined by the hashes), UDP encapsulation will not occur and normal ESP will take place. The NAT Traversal option is to enable the NAT discovery process to take place, not to force UDP encapsulation to occur.