FW Ha Cluster disconnected from Panorama

[AndreaB]

Dear all,

i m having an issue with a FW HA Cluster that is continuosly disconnecting from Panorama. 

I m able to make them connected by issuing a local commit, but after a day less or more, both the FWs are in a Disconnected state again.

There is no communication issue beetween FWs and Panorama: 

show netstat all yes numeric-hosts yes numeric-ports yes | match 172.30.0.237
tcp 0 64933 172.30.5.101:38356 172.30.0.237:3978 ESTABLISHED
tcp 0 23853 172.30.5.101:38702 172.30.0.237:3978 ESTABLISHED

172.30.0.237 -> Panorama IP address

172.30.5.101 -> FW address

Here is the evidence from ms.log:

2023-09-26 02:08:42.166 +0200 MS: peer watch. sock=28 curtime=413896 recvtime=413835 errcount=1
2023-09-26 02:09:42.166 +0200 MS: peer watch. sock=28 curtime=413956 recvtime=413835 errcount=2
2023-09-26 02:10:42.166 +0200 Error: pan_evtmgr_client_check_action(evtmgr_client_action.c:817): MS: peer timed out. sock=28 curtime=414016 recvtime=413835 errcount=3
2023-09-26 02:11:12.169 +0200 cmsa: agent index=0
2023-09-26 02:11:12.169 +0200 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
2023-09-26 02:11:12.183 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:11:12.194 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.194 +0200 Warning: _build_sc3_ssl_conn_ctx(src_panos/cms_agent.c:2042): SC3A: created context?
2023-09-26 02:11:12.194 +0200 Warning: pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2359): SC3A: client using SNI: '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:11:12.195 +0200 SC3: SNI set to '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:11:12.436 +0200 COMM: connection established. sock=28 remote ip=172.30.0.237 port=3978 local port=52962
2023-09-26 02:11:12.436 +0200 cms agent: Pre. send buffer limit=461312. s=28
2023-09-26 02:11:12.436 +0200 cms agent: Post. send buffer limit=425984. s=28
2023-09-26 02:11:12.436 +0200 Warning: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:925): SC3A: client will use sni:'3f90f814-ae99-4f76-a104-4f6f7f100558' and ccn:'487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:11:12.437 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:11:12.448 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.448 +0200 cmsa: client will use SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.474 +0200 SC3: Cert-Verify (1) /CN=3f90f814-ae99-4f76-a104-4f6f7f100558 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.474 +0200 SC3: using SC3 CA cert for validation
2023-09-26 02:11:12.477 +0200 SC3: Cert-Verify (0) /CN=2ad5dcbb-c543-4b34-9715-ee213c806e12/OU=000702469466 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.498 +0200 panorama agent: ssl channel established. sock=28 ssl=0x55c6a4032a40
2023-09-26 02:11:12.498 +0200 Device info set to panorama
2023-09-26 02:12:14.166 +0200 MS: peer watch. sock=28 curtime=414108 recvtime=414046 errcount=1
2023-09-26 02:12:41.364 +0200 update client device info, n_entries=1 op=2
2023-09-26 02:12:41.364 +0200 Device info updated for client id 1003311 device_registered no
2023-09-26 02:13:11.369 +0200 cmsa: agent index=0
2023-09-26 02:13:11.369 +0200 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
2023-09-26 02:13:11.370 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:13:11.382 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.382 +0200 Warning: _build_sc3_ssl_conn_ctx(src_panos/cms_agent.c:2042): SC3A: created context?
2023-09-26 02:13:11.382 +0200 Warning: pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2359): SC3A: client using SNI: '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:13:11.383 +0200 SC3: SNI set to '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:13:11.636 +0200 COMM: connection established. sock=28 remote ip=172.30.0.237 port=3978 local port=53344
2023-09-26 02:13:11.636 +0200 cms agent: Pre. send buffer limit=461312. s=28
2023-09-26 02:13:11.636 +0200 cms agent: Post. send buffer limit=425984. s=28
2023-09-26 02:13:11.636 +0200 Warning: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:925): SC3A: client will use sni:'3f90f814-ae99-4f76-a104-4f6f7f100558' and ccn:'487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:13:11.637 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:13:11.648 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.648 +0200 cmsa: client will use SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.666 +0200 SC3: Cert-Verify (1) /CN=3f90f814-ae99-4f76-a104-4f6f7f100558 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.666 +0200 SC3: using SC3 CA cert for validation
2023-09-26 02:13:11.669 +0200 SC3: Cert-Verify (0) /CN=2ad5dcbb-c543-4b34-9715-ee213c806e12/OU=000702469466 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.689 +0200 panorama agent: ssl channel established. sock=28 ssl=0x55c6a4033400
2023-09-26 02:13:11.689 +0200 Device info set to panorama

It doesn't look like one of the cases already discussed here on the community.

Some of you have already seen a similar case? Any suggestion for the possible root cause?

Thanks in advance

[JayGolf]

Hi @AndreaB ,

Are you using custom certificates for authentication between Panorama and the firewall? In the captured time, it looks like the connection to the peer (Panorama) times out. Then, the firewall and panorama are able to re-establish an SSL connection again. If the connection stays on for a day or so then times out, I'm leaning to believe there might be a network connectivity issue. I would verify that there are no intermittent network issues that could cause "2023-09-26 02:10:42.166 +0200 Error: pan_evtmgr_client_check_action(evtmgr_client_action.c:817): MS: peer timed out. sock=28 curtime=414016 recvtime=413835 errcount=3". If you continue to have issues, please reach out to TAC and create a support case for this issue. 

[Elizabethwayne]

Hey there, have you tried rebooting both firewalls?