IPSec Dynamic Peer VPN, failure to send traffic over attached tunnel interface

[Adrian_Jensen]

Is anyone aware of a known issue with sending traffic over an IPSec tunnel interface when using multiple dynamic peers with FQDN (host) peer identification?

I have multiple existing branch locations connected to the PA with IKEv2 IPSec tunnels using dynamic FQDN (host) peer identification from Cisco branch routers. Up to now it has worked fine, no problems re-establishing traffic after outages/reboots. After adding a new IPSec yesterday for a new branch, an old branch location broke overnight (forced new IKE key after upstream router reboot, IKE/IPSec rekey worked fine up till then). The correct IPSec gateway/tunnel comes up immediately (phase 1/2 complete normally, no errors), but the attached tunnel will not pass traffic. Both sides show outbound packets but no inbound packets received over the IPSec tunnel. I believe the PaloAlto is dropping the traffic in both directions on the tunnel (or trying to send through the wrong tunnel), but nothing shows as dropped or misrouted in logging.

I was unable to get the failed site to pass traffic until I disabled the new branch IKE/IPSec. Both IKE gateways have unique FQDNs. Among all the branch locations, the only difference is that the failing location is set as Passive/NAT as it is behind a CGN.

mp_log ikemgr.log shows the initial connection as matching the new branch setup (expected as Branch_07 is the first available dynamic peer match), followed by switching to the expected Branch_22 setup and tunnel based on received FQDN and attaching to the correct tunnel:

: received IKE request xx.xx.xx.xx[37512] to xx.xx.xx.xx[500], found IKE gateway Branch_07
: ====> IKEv2 IKE SA NEGOTIATION STARTED AS RESPONDER, non-rekey; gateway Branch_07 <====
  ====> Initiated SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] SPI:xxx:zzz SN:49662 <====
...
: xx.xx.xx.xx[37512] - xx.xx.xx.xx[500]:0x555555zzz received ID_I (type fqdn [abc.def.ghi]) matches IKE gateway Branch_22
: [IKE SA hashtbl update] from 2 to 10 (new gw: Branch_22).
: xx.xx.xx.xx[500] - xx.xx.xx.xx[37512]:yyy authentication result: success
...
: ====> IKEv2 CHILD SA NEGOTIATION STARTED AS RESPONDER, non-rekey; gateway Branch_22 <====
  ====> Initiated SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] message id:0x00000001 parent SN:49662 <====
...
: ====> IPSEC KEY INSTALLATION SUCCEEDED; tunnel Branch_22 <====
====> Installed SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] SPI:yyy/zzz lifetime 3600 Sec lifesize unlimited <====
...
: ====> IKEv2 IKE SA NEGOTIATION SUCCEEDED AS RESPONDER, non-rekey; gateway Branch_22 <====
  ====> Established SA: xx.xx.xx.xx[500]-xx.xx.xx.xx[37512] SPI:yyy:zzz SN:49662 lifetime 28800 Sec <====