IPSEC Tunnel to Azure - Odd pattern

[JakeHarris]

We received a report of some connectivity issues with an IPSEC tunnel between a Palo 5220 (9.1.8) and Azure VNG

Looking at this deeper, we see an odd rekey pattern happening with the IPSEC Rekey.  Every 4th rekey is a non-rekey and occurs short.  Can anyone help us understand what could possibly be causing this?  Its happening only on Azure VPN tunnels (multiple).

In the detail log of this rekey we see this:  Note: the first message, this is only seen on a short rekey (non-rekey)

{ 3: }: test-vpn-azure: IKEv2 SA test initiate start.
{ 3: }: ====> IKEv2 IKE SA NEGOTIATION STARTED AS INITIATOR, non-rekey; gateway test-vpn-azure <====
====> Initiated SA: x.x.x.x[500]-x.x.x.x[500] SPI:c8d0792b57f54007:0000000000000000 SN:1382 <====
{ 3: }: x.x.x.x[500] - x.x.x.x[500]:0x7f80364df7d0 ignoring notification payload (type NAT_DETECTION_SOURCE_IP) inside unauthenticated response
{ 3: }: x.x.x.x[500] - x.x.x.x[500]:0x7f80364df7d0 ignoring notification payload (type NAT_DETECTION_DESTINATION_IP) inside unauthenticated response
{ 3: }: x.x.x.x[500] - x.x.x.x[500]:0x7f80364df7d0 vendor id payload ignored
{ 3: }: x.x.x.x[500] - x.x.x.x[500]:0x7f80364df7d0 vendor id payload ignored
{ 3: }: ====> IKEv2 CHILD SA NEGOTIATION STARTED AS INITIATOR, non-rekey; gateway test-vpn-azure <====
====> Initiated SA: x.x.x.x[500]-x.x.x.x[500] message id:0x00000000 parent SN:1382 <====
{ 3: }: x.x.x.x[500] - x.x.x.x[500]:0x7f8020018720 authentication result: success
{ 3: 42}: SADB_UPDATE proto=255 x.x.x.x[500]=>x.x.x.x[500] ESP tunl spi 0x9683CEAB auth=NON-AUTH enc=AES256-GCM16/36 lifetime soft 22932/0 hard 27000/0
{ 3: 42}: SADB_ADD proto=255 x.x.x.x[500]=>x.x.x.x[500] ESP tunl spi 0x2CE69680 auth=NON-AUTH enc=AES256-GCM16/36 lifetime soft 22161/0 hard 27000/0
{ 3: 42}: ====> IPSEC KEY INSTALLATION SUCCEEDED; tunnel test-vpn-azure <====
====> Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x9683CEAB/0x2CE69680 lifetime 27000 Sec lifesize unlimited <====
{ 3: 42}: ====> IKEv2 CHILD SA NEGOTIATION SUCCEEDED AS INITIATOR, non-rekey; tunnel test-vpn-azure <====
====> Established SA: x.x.x.x[500]-x.x.x.x[500] message id:0x00000001, SPI:0x9683CEAB/0x2CE69680 parent SN:1382 <====
{ 3: }: ====> IKEv2 IKE SA NEGOTIATION SUCCEEDED AS INITIATOR, non-rekey; gateway test-vpn-azure <====
====> Established SA: x.x.x.x[500]-x.x.x.x[500] SPI:c8d0792b57f54007:0a2d7efeeb1193e1 SN:1382 lifetime 28800 Sec <====
{ 3: }: === reauth: SA 1363 is to be replaced ===
{ 3: 42}: child SA has been replaced by 5748235
{ 3: 42}: child SA has been replaced, skip
{ 3: }: child SA state EXPIRED, skip
{ 3: }: no more child SA to be renegotiated
{ 3: }: remove duplicate IKE SA, sn 1363
{ 3: }: received DELETE payload, protocol ESP, num of SPI: 1 IKE SA state ESTABLISHED
{ 3: }: delete proto ESP spi 0x6EE318D4
{ 3: 42}: ====> IPSEC KEY DELETED; tunnel test-vpn-azure <====
====> Deleted SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xCD0C0EC3/0x6EE318D4 <====
{ 3: 42}: SADB_DELETE proto=255 src=x.x.x.x[0] dst=x.x.x.x[0] ESP spi=0xCD0C0EC3
{ 3: 42}: SPI 9683CEAB inserted by IKE-SA rekey, return 0 0.
{ 3: 42}: SPI CD0C0EC3 removed by IKE SA delete, return 0 0.
{ 3: }: received DELETE payload, gateway test-vpn-azure SA state ESTABLISHED, SPI fdbd7849a0b8df4c:f1ddca2cbbb7576c
{ 3: }: x.x.x.x[500] - x.x.x.x[500]:(nil) closing IKEv2 SA test-vpn-azure:1363, code 7

Time since rekey	Log time	subtype	tunnelname	message
06:37:48	4/2/2021 19:59	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x81034CD1/0x926E541E lifetime 27000 Sec lifesize unlimited.
06:18:54	4/3/2021 2:18	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xCD0C0EC3/0x6EE318D4 lifetime 27000 Sec lifesize unlimited.
00:54:26	4/3/2021 3:12	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x9683CEAB/0x2CE69680 lifetime 27000 Sec lifesize unlimited.
06:09:19	4/3/2021 9:22	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xCF69DCCE/0xB5BEF37C lifetime 27000 Sec lifesize unlimited.
06:26:33	4/3/2021 15:48	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x932AE983/0x934683A0 lifetime 27000 Sec lifesize unlimited.
06:01:12	4/3/2021 21:49	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDA550800/0xD174AA42 lifetime 27000 Sec lifesize unlimited.
01:53:24	4/3/2021 23:43	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xB91F7B2C/0xC19A9436 lifetime 27000 Sec lifesize unlimited.
06:29:24	4/4/2021 6:12	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xD4FD6806/0x0F125AE4 lifetime 27000 Sec lifesize unlimited.
06:42:09	4/4/2021 12:55	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDA33BD67/0xEF69E2C0 lifetime 27000 Sec lifesize unlimited.
06:43:57	4/4/2021 19:38	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xAA2F1596/0xD55198EE lifetime 27000 Sec lifesize unlimited.
00:45:09	4/4/2021 20:24	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xCD989A2B/0xCD6926FC lifetime 27000 Sec lifesize unlimited.
06:21:45	4/5/2021 2:45	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x9E19B31B/0xE4A767CE lifetime 27000 Sec lifesize unlimited.
06:32:15	4/5/2021 9:18	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x926F8224/0xD2BC38B4 lifetime 27000 Sec lifesize unlimited.
06:29:51	4/5/2021 15:47	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x85C728D3/0xC1B881B6 lifetime 27000 Sec lifesize unlimited.
00:57:18	4/5/2021 16:45	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xF749CCFC/0x5AEE5E60 lifetime 27000 Sec lifesize unlimited.
06:29:33	4/5/2021 23:14	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDBAADBE9/0x786C75AE lifetime 27000 Sec lifesize unlimited.
06:07:57	4/6/2021 5:22	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xFE0969C4/0xE5AF7E2E lifetime 27000 Sec lifesize unlimited.
06:17:06	4/6/2021 11:39	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xFCAC5175/0xE067D15C lifetime 27000 Sec lifesize unlimited.
01:36:26	4/6/2021 13:16	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xB3725065/0x83C35C10 lifetime 27000 Sec lifesize unlimited.
06:21:01	4/6/2021 19:37	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x96A719F1/0x5633637E lifetime 27000 Sec lifesize unlimited.
06:26:33	4/7/2021 2:03	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xF6124B6D/0xD2B1BEF8 lifetime 27000 Sec lifesize unlimited.
06:08:50	4/7/2021 8:12	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDDE237D5/0x9907DC8E lifetime 27000 Sec lifesize unlimited.
01:32:25	4/7/2021 9:45	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDD8D0279/0x204E5CC0 lifetime 27000 Sec lifesize unlimited.
06:44:41	4/7/2021 16:29	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xEA6222A8/0x5CFC83CE lifetime 27000 Sec lifesize unlimited.
06:37:04	4/7/2021 23:06	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xAAB91393/0x51355638 lifetime 27000 Sec lifesize unlimited.
06:37:12	4/8/2021 5:44	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xB315DCDF/0x5E3ED9A8 lifetime 27000 Sec lifesize unlimited.
00:32:32	4/8/2021 6:16	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x86E54C0C/0xB4E6AB98 lifetime 27000 Sec lifesize unlimited.
06:33:46	4/8/2021 12:50	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDBC676F4/0xB78D532C lifetime 27000 Sec lifesize unlimited.
06:27:27	4/8/2021 19:17	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xE2035E0B/0x13C40192 lifetime 27000 Sec lifesize unlimited.
06:34:12	4/9/2021 1:52	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xDC4328BB/0x5E4B110C lifetime 27000 Sec lifesize unlimited.
00:34:57	4/9/2021 2:26	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0xA3148E6C/0x88754888 lifetime 27000 Sec lifesize unlimited.
06:41:41	4/9/2021 9:08	ipsec-key-install	Test-VPN-Azure	IPSec key installed. Installed SA: x.x.x.x[500]-x.x.x.x[500] SPI:0x8A0888AC/0x6740FE62 lifetime 27000 Sec lifesize unlimited.