FW Ha Cluster disconnected from Panorama

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

FW Ha Cluster disconnected from Panorama

L0 Member

Dear all,

 

i m having an issue with a FW HA Cluster that is continuosly disconnecting from Panorama. 

I m able to make them connected by issuing a local commit, but after a day less or more, both the FWs are in a Disconnected state again.

 

There is no communication issue beetween FWs and Panorama: 

 

show netstat all yes numeric-hosts yes numeric-ports yes | match 172.30.0.237
tcp 0 64933 172.30.5.101:38356 172.30.0.237:3978 ESTABLISHED
tcp 0 23853 172.30.5.101:38702 172.30.0.237:3978 ESTABLISHED

 

172.30.0.237 -> Panorama IP address

172.30.5.101 -> FW address

 

Here is the evidence from ms.log:


2023-09-26 02:08:42.166 +0200 MS: peer watch. sock=28 curtime=413896 recvtime=413835 errcount=1
2023-09-26 02:09:42.166 +0200 MS: peer watch. sock=28 curtime=413956 recvtime=413835 errcount=2
2023-09-26 02:10:42.166 +0200 Error: pan_evtmgr_client_check_action(evtmgr_client_action.c:817): MS: peer timed out. sock=28 curtime=414016 recvtime=413835 errcount=3
2023-09-26 02:11:12.169 +0200 cmsa: agent index=0
2023-09-26 02:11:12.169 +0200 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
2023-09-26 02:11:12.183 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:11:12.194 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.194 +0200 Warning: _build_sc3_ssl_conn_ctx(src_panos/cms_agent.c:2042): SC3A: created context?
2023-09-26 02:11:12.194 +0200 Warning: pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2359): SC3A: client using SNI: '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:11:12.195 +0200 SC3: SNI set to '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:11:12.436 +0200 COMM: connection established. sock=28 remote ip=172.30.0.237 port=3978 local port=52962
2023-09-26 02:11:12.436 +0200 cms agent: Pre. send buffer limit=461312. s=28
2023-09-26 02:11:12.436 +0200 cms agent: Post. send buffer limit=425984. s=28
2023-09-26 02:11:12.436 +0200 Warning: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:925): SC3A: client will use sni:'3f90f814-ae99-4f76-a104-4f6f7f100558' and ccn:'487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:11:12.437 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:11:12.448 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.448 +0200 cmsa: client will use SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.474 +0200 SC3: Cert-Verify (1) /CN=3f90f814-ae99-4f76-a104-4f6f7f100558 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.474 +0200 SC3: using SC3 CA cert for validation
2023-09-26 02:11:12.477 +0200 SC3: Cert-Verify (0) /CN=2ad5dcbb-c543-4b34-9715-ee213c806e12/OU=000702469466 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:11:12.498 +0200 panorama agent: ssl channel established. sock=28 ssl=0x55c6a4032a40
2023-09-26 02:11:12.498 +0200 Device info set to panorama
2023-09-26 02:12:14.166 +0200 MS: peer watch. sock=28 curtime=414108 recvtime=414046 errcount=1
2023-09-26 02:12:41.364 +0200 update client device info, n_entries=1 op=2
2023-09-26 02:12:41.364 +0200 Device info updated for client id 1003311 device_registered no
2023-09-26 02:13:11.369 +0200 cmsa: agent index=0
2023-09-26 02:13:11.369 +0200 [Secure conn] Secure channel for Firewall to panorama communication not enabled for secure conn.
2023-09-26 02:13:11.370 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:13:11.382 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.382 +0200 Warning: _build_sc3_ssl_conn_ctx(src_panos/cms_agent.c:2042): SC3A: created context?
2023-09-26 02:13:11.382 +0200 Warning: pan_cmsa_mgmt_assign_ssl_ctx(src_panos/cms_agent.c:2359): SC3A: client using SNI: '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:13:11.383 +0200 SC3: SNI set to '3f90f814-ae99-4f76-a104-4f6f7f100558'
2023-09-26 02:13:11.636 +0200 COMM: connection established. sock=28 remote ip=172.30.0.237 port=3978 local port=53344
2023-09-26 02:13:11.636 +0200 cms agent: Pre. send buffer limit=461312. s=28
2023-09-26 02:13:11.636 +0200 cms agent: Post. send buffer limit=425984. s=28
2023-09-26 02:13:11.636 +0200 Warning: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:925): SC3A: client will use sni:'3f90f814-ae99-4f76-a104-4f6f7f100558' and ccn:'487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:13:11.637 +0200 SC3: CA: '3f90f814-ae99-4f76-a104-4f6f7f100558', CC/CSR: '487af64e-1a47-410a-9bd8-e72cd63f15bc'
2023-09-26 02:13:11.648 +0200 SC3: context initialized using SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.648 +0200 cmsa: client will use SNI: 3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.666 +0200 SC3: Cert-Verify (1) /CN=3f90f814-ae99-4f76-a104-4f6f7f100558 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.666 +0200 SC3: using SC3 CA cert for validation
2023-09-26 02:13:11.669 +0200 SC3: Cert-Verify (0) /CN=2ad5dcbb-c543-4b34-9715-ee213c806e12/OU=000702469466 :: /CN=3f90f814-ae99-4f76-a104-4f6f7f100558
2023-09-26 02:13:11.689 +0200 panorama agent: ssl channel established. sock=28 ssl=0x55c6a4033400
2023-09-26 02:13:11.689 +0200 Device info set to panorama

 

It doesn't look like one of the cases already discussed here on the community.

Some of you have already seen a similar case? Any suggestion for the possible root cause?

 

Thanks in advance

2 REPLIES 2

Community Team Member

Hi @AndreaB ,

 

Are you using custom certificates for authentication between Panorama and the firewall? In the captured time, it looks like the connection to the peer (Panorama) times out. Then, the firewall and panorama are able to re-establish an SSL connection again. If the connection stays on for a day or so then times out, I'm leaning to believe there might be a network connectivity issue. I would verify that there are no intermittent network issues that could cause "2023-09-26 02:10:42.166 +0200 Error: pan_evtmgr_client_check_action(evtmgr_client_action.c:817): MS: peer timed out. sock=28 curtime=414016 recvtime=413835 errcount=3". If you continue to have issues, please reach out to TAC and create a support case for this issue. 

LIVEcommunity team member
Stay Secure,
Jay
Don't forget to Like items if a post is helpful to you!

Please help out other users and “Accept as Solution” if a post helps solve your problem !

Read more about how and why to accept solutions.

L0 Member

Hey there, have you tried rebooting both firewalls?

Liza
  • 486 Views
  • 2 replies
  • 1 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!