Firewall Disconnected from Secondary Panorama

alan-griffiths — Fri, 07 Oct 2022 13:29:49 GMT

Added some new firewalls to a Panorama HA pair and one of the devices is disconnected from the secondary Panorama.

admin@intra-az1> show panorama-status Panorama Server 1 : 10.201.24.12 Connected : yes HA state : Active Panorama Server 2 : 10.201.25.12 Connected : no HA state : disconnected

Running tcpdump I can see traffic is passing between the device and the Panorama

2:56:14.150509 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4362:4431, ack 1, win 296, options [nop,nop,TS val 3086412655 ecr 1690590683], length 69 12:56:14.151873 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4431, win 379, options [nop,nop,TS val 1690591348 ecr 3086412655], length 0 12:56:17.980601 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 69:138, ack 70, win 332, options [nop,nop,TS val 3086416485 ecr 1690589179], length 69 12:56:17.982715 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 70:139, ack 138, win 293, options [nop,nop,TS val 1690595179 ecr 3086416485], length 69 12:56:17.982730 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [.], ack 139, win 332, options [nop,nop,TS val 3086416487 ecr 1690595179], length 0 12:56:20.150517 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4431:4500, ack 1, win 296, options [nop,nop,TS val 3086418655 ecr 1690591348], length 69 12:56:20.151884 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4500, win 379, options [nop,nop,TS val 1690597348 ecr 3086418655], length 0 12:56:23.980629 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 138:207, ack 139, win 332, options [nop,nop,TS val 3086422485 ecr 1690595179], length 69 12:56:23.982485 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 139:208, ack 207, win 293, options [nop,nop,TS val 1690601179 ecr 3086422485], length 69 12:56:23.982511 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [.], ack 208, win 332, options [nop,nop,TS val 3086422487 ecr 1690601179], length 0 12:56:26.150520 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4500:4569, ack 1, win 296, options [nop,nop,TS val 3086424655 ecr 1690597348], length 69 12:56:26.151931 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4569, win 379, options [nop,nop,TS val 1690603348 ecr 3086424655], length 0 12:56:29.980632 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 207:276, ack 208, win 332, options [nop,nop,TS val 3086428485 ecr 1690601179], length 69 12:56:29.982366 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 208:277, ack 276, win 293, options [nop,nop,TS val 1690607179 ecr 3086428485], length 69 12:56:29.982385 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [.], ack 277, win 332, options [nop,nop,TS val 3086428486 ecr 1690607179], length 0 12:56:32.150527 IP 10.201.50.52.48026 > 10.201.25.12.pan-panorama: Flags [P.], seq 4569:4638, ack 1, win 296, options [nop,nop,TS val 3086430655 ecr 1690603348], length 69 12:56:32.151961 IP 10.201.25.12.pan-panorama > 10.201.50.52.48026: Flags [.], ack 4638, win 379, options [nop,nop,TS val 1690609349 ecr 3086430655], length 0 12:56:35.980626 IP 10.201.50.52.46264 > 10.201.25.12.pan-panorama: Flags [P.], seq 276:345, ack 277, win 332, options [nop,nop,TS val 3086434485 ecr 1690607179], length 69 12:56:35.982329 IP 10.201.25.12.pan-panorama > 10.201.50.52.46264: Flags [P.], seq 277:346, ack 345, win 293, options [nop,nop,TS val 1690613179 ecr 3086434485], length 6

From ms.log I this cycle every minute

2022-10-07 13:22:54.849 +0000 update client device info, n_entries=1 op=2 2022-10-07 13:22:54.849 +0000 Device info updated for client id 1000055 device_registered no 2022-10-07 13:23:24.850 +0000 cmsa: agent index=1 2022-10-07 13:23:24.851 +0000 Warning: sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI 2022-10-07 13:23:24.851 +0000 Warning: sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN 2022-10-07 13:23:24.851 +0000 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn. 2022-10-07 13:23:24.856 +0000 Warning: pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2353): client using default (legacy) context 2022-10-07 13:23:24.856 +0000 Warning: sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI 2022-10-07 13:23:24.856 +0000 Warning: sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN 2022-10-07 13:23:25.093 +0000 COMM: connection established. sock=29 remote ip=10.201.25.12 port=3978 local port=51960 2022-10-07 13:23:25.093 +0000 cms agent: Pre. send buffer limit=87040. s=29 2022-10-07 13:23:25.093 +0000 cms agent: Post. send buffer limit=2097152. s=29 2022-10-07 13:23:25.093 +0000 Error: cs_load_certs_ex(cs_common.c:655): keyfile not exists 2022-10-07 13:23:25.093 +0000 Error: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:883): cms agent: cs_load_certs_ex failed 2022-10-07 13:23:25.093 +0000 cmsa: client will use default context 2022-10-07 13:23:25.093 +0000 Warning: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:988): client will not use SNI 2022-10-07 13:23:25.098 +0000 panorama agent: ssl channel established. sock=29 ssl=0x555fd2a82700 2022-10-07 13:23:25.098 +0000 Device info set to panorama2 2022-10-07 13:24:54.849 +0000 update client device info, n_entries=1 op=2 2022-10-07 13:24:54.849 +0000 Device info updated for client id 1000056 device_registered no

Don't really know what else to check. I added four devices at the same time and the other three are connected fine, so don't understand what went wrong with this one.

Re: Firewall Disconnected from Secondary Panorama

alan-griffiths — Thu, 13 Oct 2022 10:00:31 GMT

Yeah, that did the trick. Bit strange though. I had to failover to the secondary to re-add the firewall. If I did it on the primary then it would just come back on the secondary as disconnected again.

Remove device on primary, commit.
Remove panorama config from device.
Failover to secondary Panorama.
Add device, commit.
Re-dd primary and secondary Panorama config on device.
Verify device is reported as connected on both primary and secondary.
Fail back to primary.

topic Firewall Disconnected from Secondary Panorama in Panorama Discussions

Firewall Disconnected from Secondary Panorama

Re: Firewall Disconnected from Secondary Panorama

Re: Firewall Disconnected from Secondary Panorama