Firewall not connecting to Panorama

[AdamHP]

Hello I have new deployed Panorama and new PA-440 Firewall.

I setup Panorama with all basic settings like IP address/netmask, default GW, DNS, it has license assigned.

Next I generated AuthKey for the firewalls with validity for 10 days and without SN specified. 

PA-440 is in remote location and has a basic WAN setup and IPSec VPN to my datacenter where panorama is. 

It has a vlan interface setup in my internal zone and set as source for every service.

I am able to ping Panorama from the PA-440 so network over VPN is working.

When I setup Panorama IP with Auth Key on the firewall and add Firewall on panorama by the Serial Number I still see PA-440 in panorama as Disconnected.

I checked the DataCenter firewall where IPSec is terminated and I can''t see in logs any blocked traffic in between these two.

Port 3978 for Panorama  is enabled in security rules and I can see some ssl traffic is passing in Datacenter over this port.

Is there something else I forgott to setup or something else I need to check in order to be able to manage this Firewall by Panorama?

[PavelK]

Thank you for reply @AdamHP

If TCP connection is not established, it seems it is failing during initial connection setup. Would it be possible to take packet capture on Panorama side: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CleECAS to see at what phase it is failing and whether any of the side is sending RST?

Kind Regards

Pavel

[AdamHP]

Hello Pavel, 

sample output of TCP dump is here(it is looping this same sequence all the time from what I saw in TCP dump)

23:08:44.614213 IP Firewall.54198 > Panorama.pan-panorama: Flags [S], seq 4083635727, win 29200, options [mss 1260,sackOK,TS val 4055216138 ecr 0,nop,wscale 7], length 0
23:08:44.614259 IP Panorama.pan-panorama > Firewall.54198: Flags [S.], seq 4178576975, ack 4083635728, win 24960, options [mss 1260,sackOK,TS val 2636905311 ecr 4055216138,nop,wscale 7], length 0
23:08:44.641051 IP Firewall.54198 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216165 ecr 2636905311], length 0
23:08:44.641107 IP Firewall.54198 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0
23:08:44.993054 IP Firewall.54204 > Panorama.pan-panorama: Flags [S], seq 177465872, win 29200, options [mss 1260,sackOK,TS val 4055216517 ecr 0,nop,wscale 7], length 0
23:08:44.993100 IP Panorama.pan-panorama > Firewall.54204: Flags [S.], seq 1639741662, ack 177465873, win 24960, options [mss 1260,sackOK,TS val 2636905690 ecr 4055216517,nop,wscale 7], length 0
23:08:45.018435 IP Firewall.54204 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216543 ecr 2636905690], length 0
23:08:45.031530 IP Firewall.54204 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0

[PavelK]

Thank you for getting packet capture @AdamHP

Based on packet capture, the Firewall is resetting the connection by setting RST flag. Based on all information that you supplied, I am not clear why this is happening. Would it be possible to look into logs on Firewall to see it can provide more details: tail lines 1000 mp-log ms.log

Kind Regards

Pavel 

[ahandoo]

Hi @AdamHP

To be double sure that the firewall is the one sending the RST and not any intermediate device, I would take a simultaneous packet capture on Firewall and Panorama. An alternate way would be to compare the TTL value (in the IP Header) of the RST with the SYN packet (if both are sent from the same host, TTL will be the same).

Regards

[AdamHP]

Hi Pavel, 

this look like the interesting part from this log as it is looping there.

2022-03-04 09:35:18.328 +0100 COMM: connection established. sock=29 remote ip=PANORAMA_IP port=3978 local port=35712
2022-03-04 09:35:18.328 +0100 cms agent: Pre. send buffer limit=46080. s=29
2022-03-04 09:35:18.328 +0100 cms agent: Post. send buffer limit=425984. s=29
2022-03-04 09:35:18.328 +0100 Error:  cs_load_certs_ex(cs_common.c:654): keyfile not exists
2022-03-04 09:35:18.328 +0100 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:876): cms agent: cs_load_certs_ex failed
2022-03-04 09:35:18.328 +0100 cmsa: client will use default context
2022-03-04 09:35:18.331 +0100 Error:  sc3_ca_exists(sc3_certs.c:221): SC3: Failed to get the current CA name.
2022-03-04 09:35:18.331 +0100 Warning:  sc3_init_sc3(sc3_utils.c:351): SC3: Failed to get the Current CC name
2022-03-04 09:35:18.331 +0100 SC3: CA: '', CC/CSR: '9469a205-8e13-46ed-879c-13d45a0ae772'
2022-03-04 09:35:18.332 +0100 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
2022-03-04 09:35:18.332 +0100 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
2022-03-04 09:35:18.341 +0100 Warning:  sc3_init_sctx(sc3_ctx.c:323): SC3: not set, skip cert loading
2022-03-04 09:35:18.341 +0100 SC3A: using SNI (from AK): 75220d86-f64a-4a64-b542-1b81b8cae893
2022-03-04 09:35:18.341 +0100 SC3A: using sc3 ctx with no cert
2022-03-04 09:35:18.342 +0100 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:1196): panorama agent: SSL connect error. sock=29 err=5

It look like it can connect which is the traffic I can see in the Datacenter Firewall and the problem is with SSL?