Firewall Disconnected from Secondary Panorama

[alan-griffiths]

Added some new firewalls to a Panorama HA pair and one of the devices is disconnected from the secondary Panorama.

admin@intra-az1> show panorama-status 

Panorama Server 1 : 10.201.24.12
    Connected     : yes
    HA state      : Active

Panorama Server 2 : 10.201.25.12
    Connected     : no
    HA state      : disconnected

Running tcpdump I can see traffic is passing between the device and the Panorama

2:56:14.150509 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4362:4431, ack 1, win 296, options [nop,nop,TS val 3086412655 ecr 1690590683], length 69
12:56:14.151873 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4431, win 379, options [nop,nop,TS val 1690591348 ecr 3086412655], length 0
12:56:17.980601 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 69:138, ack 70, win 332, options [nop,nop,TS val 3086416485 ecr 1690589179], length 69
12:56:17.982715 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 70:139, ack 138, win 293, options [nop,nop,TS val 1690595179 ecr 3086416485], length 69
12:56:17.982730 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [.], ack 139, win 332, options [nop,nop,TS val 3086416487 ecr 1690595179], length 0
12:56:20.150517 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4431:4500, ack 1, win 296, options [nop,nop,TS val 3086418655 ecr 1690591348], length 69
12:56:20.151884 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4500, win 379, options [nop,nop,TS val 1690597348 ecr 3086418655], length 0
12:56:23.980629 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 138:207, ack 139, win 332, options [nop,nop,TS val 3086422485 ecr 1690595179], length 69
12:56:23.982485 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 139:208, ack 207, win 293, options [nop,nop,TS val 1690601179 ecr 3086422485], length 69
12:56:23.982511 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [.], ack 208, win 332, options [nop,nop,TS val 3086422487 ecr 1690601179], length 0
12:56:26.150520 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4500:4569, ack 1, win 296, options [nop,nop,TS val 3086424655 ecr 1690597348], length 69
12:56:26.151931 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4569, win 379, options [nop,nop,TS val 1690603348 ecr 3086424655], length 0
12:56:29.980632 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 207:276, ack 208, win 332, options [nop,nop,TS val 3086428485 ecr 1690601179], length 69
12:56:29.982366 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 208:277, ack 276, win 293, options [nop,nop,TS val 1690607179 ecr 3086428485], length 69
12:56:29.982385 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [.], ack 277, win 332, options [nop,nop,TS val 3086428486 ecr 1690607179], length 0
12:56:32.150527 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4569:4638, ack 1, win 296, options [nop,nop,TS val 3086430655 ecr 1690603348], length 69
12:56:32.151961 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4638, win 379, options [nop,nop,TS val 1690609349 ecr 3086430655], length 0
12:56:35.980626 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 276:345, ack 277, win 332, options [nop,nop,TS val 3086434485 ecr 1690607179], length 69
12:56:35.982329 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 277:346, ack 345, win 293, options [nop,nop,TS val 1690613179 ecr 3086434485], length 6

 From ms.log I this cycle every minute

2022-10-07 13:22:54.849 +0000 update client device info, n_entries=1 op=2
2022-10-07 13:22:54.849 +0000 Device info updated for client id 1000055 device_registered no
2022-10-07 13:23:24.850 +0000 cmsa: agent index=1
2022-10-07 13:23:24.851 +0000 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
2022-10-07 13:23:24.851 +0000 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
2022-10-07 13:23:24.851 +0000 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
2022-10-07 13:23:24.856 +0000 Warning:  pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2353): client using default (legacy) context
2022-10-07 13:23:24.856 +0000 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
2022-10-07 13:23:24.856 +0000 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
2022-10-07 13:23:25.093 +0000 COMM: connection established. sock=29 remote ip=10.201.25.12 port=3978 local port=51960
2022-10-07 13:23:25.093 +0000 cms agent: Pre. send buffer limit=87040. s=29
2022-10-07 13:23:25.093 +0000 cms agent: Post. send buffer limit=2097152. s=29
2022-10-07 13:23:25.093 +0000 Error:  cs_load_certs_ex(cs_common.c:655): keyfile not exists
2022-10-07 13:23:25.093 +0000 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:883): cms agent: cs_load_certs_ex failed
2022-10-07 13:23:25.093 +0000 cmsa: client will use default context
2022-10-07 13:23:25.093 +0000 Warning:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:988): client will not use SNI
2022-10-07 13:23:25.098 +0000 panorama agent: ssl channel established. sock=29 ssl=0x555fd2a82700
2022-10-07 13:23:25.098 +0000 Device info set to panorama2
2022-10-07 13:24:54.849 +0000 update client device info, n_entries=1 op=2
2022-10-07 13:24:54.849 +0000 Device info updated for client id 1000056 device_registered no

Don't really know what else to check. I added four devices at the same time and the other three are connected fine, so don't understand what went wrong with this one.

[tabner]

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000wkjSCAQ&lang=en_US%E2%80%A...

[alan-griffiths · Accepted solution]

Yeah, that did the trick. Bit strange though. I had to failover to the secondary to re-add the firewall. If I did it on the primary then it would just come back on the secondary as disconnected again.

So

• Remove device on primary, commit.
• Remove panorama config from device.
• Failover to secondary Panorama.
• Add device, commit.
• Re-dd primary and secondary Panorama config on device.
• Verify device is reported as connected on both primary and secondary.
• Fail back to primary.