topic Re: Firewall not connecting to Panorama in Panorama Discussions

Firewall not connecting to Panorama

AdamHP — Mon, 28 Feb 2022 14:44:39 GMT

Hello I have new deployed Panorama and new PA-440 Firewall.

I setup Panorama with all basic settings like IP address/netmask, default GW, DNS, it has license assigned.

Next I generated AuthKey for the firewalls with validity for 10 days and without SN specified.

PA-440 is in remote location and has a basic WAN setup and IPSec VPN to my datacenter where panorama is.

It has a vlan interface setup in my internal zone and set as source for every service.

I am able to ping Panorama from the PA-440 so network over VPN is working.

When I setup Panorama IP with Auth Key on the firewall and add Firewall on panorama by the Serial Number I still see PA-440 in panorama as Disconnected.

I checked the DataCenter firewall where IPSec is terminated and I can''t see in logs any blocked traffic in between these two.

Port 3978 for Panorama is enabled in security rules and I can see some ssl traffic is passing in Datacenter over this port.

Is there something else I forgott to setup or something else I need to check in order to be able to manage this Firewall by Panorama?

Re: Firewall not connecting to Panorama

Mudhireddy — Mon, 28 Feb 2022 18:45:47 GMT

Hi,

Can you these given ports as per your connectivity between FWs and Panorama. And same time can you check the logs

in >monitor>loigs>system level

https://docs.paloaltonetworks.com/pan-os/10-2/pan-os-admin/firewall-administration/reference-port-number-usage/ports-used-for-panorama.html

example"

On Firewall:

show panorama status

On Panorama:

show devices all

verify both commands.

Panorama APP-ID: When the managed firewall communicates with Panorama, by default this traffic is sent over the MGT interface. Because traffic leaving the MGT interface of the firewall is not subject to a Security policy check, no additional Security policy rule configuration is necessary.

However, in some deployments, the administrator might choose to send management traffic through one of the data-plane interfaces of the firewall. In this case, remember to create a Security policy rule to allow the Panorama application. Otherwise, the firewall denies communications with Panorama.

and try to see reachability:

Use ping from the firewall or Panorama command line ping count <integer> source <IP-address> host <IP-address

and try pcap on mgmt using tcpdump

•Run tcpdump from the command line of Panorama or the firewall to capture the traffic. When you have enough data, press Ctrl+C to stop the capture.

Example: tcpdump filter “host 10.1.10.10

Re: Firewall not connecting to Panorama

AdamHP — Fri, 04 Mar 2022 09:35:45 GMT

Hi,

I am using dataplane port instead of Management as the connection to panorama is anyway over the IPSec VPN connection.

The coresponding Firewall rules are applied on both Firewalls and they are passing the traffic on this ports. Also ping works fine.

From Firewall: show panorama-status

Panorama Server 1 : 192.168.1.20 Connected : no HA state : disconnected

From Panorama: show devices all

Serial Hostname IPv4 IPv6 Connected -------------------------------------------------------------------------- 00000000XXXX no Wildfire Real-time Stream Disabled VPN Disable Mode: no Operational Mode: normal Certificate Status: Certificate subject Name: Certificate expiry at: Connected at: Custom certificate Used: Last masterkey push status: Unknown Last masterkey push timestamp: none Express mode: no Device cert present : Device cert expiry date : N/A

I run also a PCAP and I can see on panorama and also on Firewall traffic from the oposite side.

I also check all this from this checklist:

1. IP connectivity is there

2. Port 3978 and also others required for panorama are open

3. DeviceCert on Panorama installed

4. Serial number of device is correct

5. Management profile set up on this interface used for communication

6. Panorama is on version 10.2, Firewall 10.1.4

7. MTU on tunnel interface lowered

8.Time synchronized using NTP

What else I can check?

Re: Firewall not connecting to Panorama

PavelK — Wed, 02 Mar 2022 12:19:57 GMT

Thank you for supplying additional information @AdamHP

Could you check from Panorama's CLI whether TCP connection is established by: show netstat numeric yes | match 3978 ?

Could you check status of Panorama's certificate from browser: https://<panorama ip>:3978 ?

Could you also check log on Panorama side from CLI: tail lines 500 mp-log configd.log ? Search events corresponding with Firewall's Serial Number.

Kind Regards

Pavel

Re: Firewall not connecting to Panorama

AdamHP — Wed, 02 Mar 2022 20:36:18 GMT

Hello Pavel,

1. netstat is showing no match, so no TCP connection on this port is established.

2. When I access panorama IP on port 3978 I will get window to select which certificate I want use to authenticate myself

After I click on OK, I see that panorama is using self-signed cert issued to localhost

3. In log only entry like this was found with the serial of the FW

2022-03-02 15:22:45.032 +0100 String is <devices> <entry name="000000000001"/> <entry name="000000000000"/> </devices> 2022-03-02 15:22:45.032 +0100 After str: <devices> <entry name="000000000001"/> <entry name="000000000000"/> </devices>

Re: Firewall not connecting to Panorama

PavelK — Wed, 02 Mar 2022 21:55:55 GMT

Thank you for reply @AdamHP

If TCP connection is not established, it seems it is failing during initial connection setup. Would it be possible to take packet capture on Panorama side: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CleECAS to see at what phase it is failing and whether any of the side is sending RST?

Kind Regards

Pavel

Re: Firewall not connecting to Panorama

AdamHP — Wed, 02 Mar 2022 22:32:17 GMT

Hello Pavel,

sample output of TCP dump is here(it is looping this same sequence all the time from what I saw in TCP dump)

23:08:44.614213 IP Firewall.54198 > Panorama.pan-panorama: Flags [S], seq 4083635727, win 29200, options [mss 1260,sackOK,TS val 4055216138 ecr 0,nop,wscale 7], length 0 23:08:44.614259 IP Panorama.pan-panorama > Firewall.54198: Flags [S.], seq 4178576975, ack 4083635728, win 24960, options [mss 1260,sackOK,TS val 2636905311 ecr 4055216138,nop,wscale 7], length 0 23:08:44.641051 IP Firewall.54198 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216165 ecr 2636905311], length 0 23:08:44.641107 IP Firewall.54198 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0 23:08:44.993054 IP Firewall.54204 > Panorama.pan-panorama: Flags [S], seq 177465872, win 29200, options [mss 1260,sackOK,TS val 4055216517 ecr 0,nop,wscale 7], length 0 23:08:44.993100 IP Panorama.pan-panorama > Firewall.54204: Flags [S.], seq 1639741662, ack 177465873, win 24960, options [mss 1260,sackOK,TS val 2636905690 ecr 4055216517,nop,wscale 7], length 0 23:08:45.018435 IP Firewall.54204 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216543 ecr 2636905690], length 0 23:08:45.031530 IP Firewall.54204 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0

Re: Firewall not connecting to Panorama

PavelK — Fri, 04 Mar 2022 00:57:26 GMT

Thank you for getting packet capture @AdamHP

Based on packet capture, the Firewall is resetting the connection by setting RST flag. Based on all information that you supplied, I am not clear why this is happening. Would it be possible to look into logs on Firewall to see it can provide more details: tail lines 1000 mp-log ms.log

Kind Regards

Pavel

Re: Firewall not connecting to Panorama

ahandoo — Fri, 04 Mar 2022 03:42:51 GMT

Hi @AdamHP

To be double sure that the firewall is the one sending the RST and not any intermediate device, I would take a simultaneous packet capture on Firewall and Panorama. An alternate way would be to compare the TTL value (in the IP Header) of the RST with the SYN packet (if both are sent from the same host, TTL will be the same).

Regards

Re: Firewall not connecting to Panorama

AdamHP — Fri, 04 Mar 2022 09:40:43 GMT

Hi Pavel,

this look like the interesting part from this log as it is looping there.

2022-03-04 09:35:18.328 +0100 COMM: connection established. sock=29 remote ip=PANORAMA_IP port=3978 local port=35712 2022-03-04 09:35:18.328 +0100 cms agent: Pre. send buffer limit=46080. s=29 2022-03-04 09:35:18.328 +0100 cms agent: Post. send buffer limit=425984. s=29 2022-03-04 09:35:18.328 +0100 Error: cs_load_certs_ex(cs_common.c:654): keyfile not exists 2022-03-04 09:35:18.328 +0100 Error: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:876): cms agent: cs_load_certs_ex failed 2022-03-04 09:35:18.328 +0100 cmsa: client will use default context 2022-03-04 09:35:18.331 +0100 Error: sc3_ca_exists(sc3_certs.c:221): SC3: Failed to get the current CA name. 2022-03-04 09:35:18.331 +0100 Warning: sc3_init_sc3(sc3_utils.c:351): SC3: Failed to get the Current CC name 2022-03-04 09:35:18.331 +0100 SC3: CA: '', CC/CSR: '9469a205-8e13-46ed-879c-13d45a0ae772' 2022-03-04 09:35:18.332 +0100 Warning: sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI 2022-03-04 09:35:18.332 +0100 Warning: sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN 2022-03-04 09:35:18.341 +0100 Warning: sc3_init_sctx(sc3_ctx.c:323): SC3: not set, skip cert loading 2022-03-04 09:35:18.341 +0100 SC3A: using SNI (from AK): 75220d86-f64a-4a64-b542-1b81b8cae893 2022-03-04 09:35:18.341 +0100 SC3A: using sc3 ctx with no cert 2022-03-04 09:35:18.342 +0100 Error: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:1196): panorama agent: SSL connect error. sock=29 err=5

It look like it can connect which is the traffic I can see in the Datacenter Firewall and the problem is with SSL?

Re: Firewall not connecting to Panorama

PavelK — Sat, 05 Mar 2022 22:36:09 GMT

Thank you for reply and getting logs @AdamHP

The Error 5 is "SSL verification failure". Unless you configured mutual SSL authentication, only Panorama has to present a certificate: https://docs.paloaltonetworks.com/panorama/10-2/panorama-admin/set-up-panorama/set-up-authentication-using-custom-certificates/how-are-ssltls-connections-mutually-authenticated.html

Would it be possible one more time to check the certificate on Panorama? The certificate is self signed, however could you confirm: valid to, issuer, subject,...?

Kind Regards

Pavel

Re: Firewall not connecting to Panorama

AdamHP — Mon, 07 Mar 2022 09:43:34 GMT

Hi Pavel,

if you mean cert on https port 3978, I am getting this.

It is issued to my panorama IP address.

Re: Firewall not connecting to Panorama

shaggy63 — Thu, 10 Mar 2022 15:52:03 GMT

Check under Monitor -> Session Browser under each firewall it flows through, even the source firewall.

Had the same problem and it turns out Panorama traffic won't match a catch all, it must specifically be application Panorama.

Re: Firewall not connecting to Panorama

AdamHP — Fri, 11 Mar 2022 16:31:58 GMT

Thank you all for help, it turns out that on source firewall PA-440 I allow traffic for Panorama Application on its default port 3978, but in firewall monitor I found that the flow is recognized as ssl on port 3978 and this was blocked.
I was thinking that once there is Panorama app it will be match and didn't check this, but I was wrong. 🙂
I had simmilar issue today on another firewall in Data Center where we had rule for WinRM (microsoft-remote-management), which was working fine before but now after some updates it is recognized as web-browsing on the WinRM port 5985.
So the solution is to not trust the Palo Alto application matching and always check the flows.

Re: Firewall not connecting to Panorama

AriKukkonen — Wed, 30 Mar 2022 22:09:11 GMT

I have also PA-440 on 10.2 and Panorama on 10.2. And same issue, FW disconnecting. I think this is a bug with 10.2

Re: Firewall not connecting to Panorama

JasonCoen — Fri, 13 May 2022 00:18:31 GMT

We recently experienced the same problem and only saw it on devices 10.1 and up. Last week or two we added devices that were 9.1.0-h3 and traffic matched on Panorama / SSL w/application default no problem. This week with 10.1 we had to add a specific match for SSL on service port 3978 to get them to connect.

Re: Firewall not connecting to Panorama

YEmreSeven — Wed, 18 May 2022 08:31:57 GMT

I faced similar issue recently. Backup firewall looked disconnected on Panorama v10.2. And traffic should be flow over active firewall's IPsec. If you see "TCP session closed via injecting RST" on Palo Alto counter, try to change your security rule with any application and tcp/3978 on service and no security profile. I hope it will solve your problem. It was worked for me.

Re: Firewall not connecting to Panorama

alterioc — Fri, 29 Jul 2022 18:35:36 GMT

I had a similar issue although I am just using the MGMT interface to connect to Panorama. I had the problem on a PA-820 I got as an RMA, and also on Palo VMs. This problem is caused by the new Panorama Device Registration Auth Key. We're running PAN-OS 10.1.

.
tail follow yes mp-log ms.log on Panorama shows a bunch of SC3 errors like "keyfile not exists", "bad certificate", "Failed to get the current CA name", "Failed to get the Current CC name", "failed to get SNI", "failed to get CCN".

tcpdump filter "port 3978" on the firewall followed by view-pcap mgmt-pcap mgmt.pcap shows the device communicating with Panorama but the device sends a RST.

Our solution was to reset sc3.

We had to do it both on the device and on Panorama to get things to work.

Palo doesn't recommend doing it on Panorama but we couldn't get it working until we did that.

.
Start by resetting sc3 on the device as shown in the three steps below.

1. On the cli of the firewall

show system info (copy the s/n for step 2)

request sc3 reset (reply y to the prompt)
debug software restart process management-server
(wait for the management-server process to come back up)

2. On Panorama cli:
clear device-status deviceid <device s/n>

3. Reconnect to the firewall cli and do:
request authkey set <authkey>
(the authkey is on Panorama, Panorama tab, on the left pane near the bottom, "Device Registration Auth Key". If no key appears, click Add to create a new one. (I just gave it a name and specfied 1 day lifetime.) Then copy/paste it into the command above.

It may take a minute or so and in some cases these steps may need to be done twice. But, if this does not work you may need to do the "request sc3 reset" and "debug software restart management-server" on Panorama (not recommended). Once the management-server process is back up, log into the Panorama UI, delete any device reg auth key and generate a new one. Then repeat the 3 steps above using the new Device Auth Reg Key.

Re: Firewall not connecting to Panorama

RonCarmack — Wed, 17 Aug 2022 12:14:18 GMT

Thanks for your post. I performed this similar fix on a firewall/Panorama and it resolved the issue. I was seeing the SC3 and bad certificate errors as well.

Thanks

Re: Firewall not connecting to Panorama

walter2020 — Fri, 07 Oct 2022 04:59:25 GMT

did you find a fix for this issue? I am having the same issue