Firewall not connecting to Panorama

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Firewall not connecting to Panorama

L1 Bithead

Hello I have new deployed Panorama and new PA-440 Firewall.

I setup Panorama with all basic settings like IP address/netmask, default GW, DNS, it has license assigned.

Next I generated AuthKey for the firewalls with validity for 10 days and without SN specified. 

 

PA-440 is in remote location and has a basic WAN setup and IPSec VPN to my datacenter where panorama is. 

It has a vlan interface setup in my internal zone and set as source for every service.

I am able to ping Panorama from the PA-440 so network over VPN is working.

When I setup Panorama IP with Auth Key on the firewall and add Firewall on panorama by the Serial Number I still see PA-440 in panorama as Disconnected.

I checked the DataCenter firewall where IPSec is terminated and I can''t see in logs any blocked traffic in between these two.

Port 3978 for Panorama  is enabled in security rules and I can see some ssl traffic is passing in Datacenter over this port.

Is there something else I forgott to setup or something else I need to check in order to be able to manage this Firewall by Panorama?

20 REPLIES 20

L3 Networker

Hi,

 

Can you these given ports as per your connectivity between FWs and Panorama. And same time can you check the logs

in >monitor>loigs>system level

https://docs.paloaltonetworks.com/pan-os/10-2/pan-os-admin/firewall-administration/reference-port-nu...

 

example"

Mudhireddy_0-1646073396693.png

On Firewall:

show panorama status

On Panorama:

show devices all

 

verify both commands.

 

Panorama APP-ID: When the managed firewall communicates with Panorama, by default this traffic is sent over the MGT interface. Because traffic leaving the MGT interface of the firewall is not subject to a Security policy check, no additional Security policy rule configuration is necessary.

However, in some deployments, the administrator might choose to send management traffic through one of the data-plane interfaces of the firewall. In this case, remember to create a Security policy rule to allow the Panorama application. Otherwise, the firewall denies communications with Panorama.

 

Mudhireddy_1-1646073541341.png

and try to see reachability:

Use ping from the firewall or Panorama command line ping count <integer> source <IP-address> host <IP-address

 

and try pcap on mgmt using tcpdump

•Run tcpdump from the command line of Panorama or the firewall to capture the traffic. When you have enough data, press Ctrl+C to stop the capture.

Example: tcpdump filter “host 10.1.10.10

 

 

Best Regards,
Suresh

Hi, 

I am using dataplane port instead of Management as the connection to panorama is anyway over the IPSec VPN connection. 

The coresponding Firewall rules are applied on both Firewalls and they are passing the traffic on this ports. Also ping works fine.

From Firewall: show panorama-status

 

Panorama Server 1 : 192.168.1.20
    Connected     : no
    HA state      : disconnected

 

From Panorama: show devices all

 

Serial                   Hostname        IPv4            IPv6                             Connected
--------------------------------------------------------------------------
00000000XXXX                                                                                     no
Wildfire Real-time Stream Disabled  VPN Disable Mode: no
  Operational Mode: normal
  Certificate Status:
  Certificate subject Name:
  Certificate expiry at:
  Connected at:
  Custom certificate Used:
  Last masterkey push status: Unknown
  Last masterkey push timestamp:  none
  Express mode: no
 Device cert present :
 Device cert expiry date : N/A

 

I run also a PCAP and I can see on panorama and also on Firewall traffic from the oposite side.

I also check all this from this checklist: 

AdamHP_0-1646164422919.png

1. IP connectivity is there

2. Port 3978 and also others required for panorama are open

3. DeviceCert on Panorama installed

4.  Serial number of device is correct

5. Management profile set up on this interface used for communication

6. Panorama is on version 10.2, Firewall 10.1.4

7. MTU on tunnel interface lowered

8.Time synchronized using NTP

 

What else I can check? 

Cyber Elite
Cyber Elite

Thank you for supplying additional information @AdamHP

 

Could you check from Panorama's CLI whether TCP connection is established by: show netstat numeric yes | match 3978 ?

Could you check status of Panorama's certificate from browser: https://<panorama ip>:3978 ?

Could you also check log on Panorama side from CLI: tail lines 500 mp-log configd.log ? Search events corresponding with Firewall's Serial Number.

 

Kind Regards

Pavel

Help the community: Like helpful comments and mark solutions.

Hello Pavel, 

1. netstat is showing no match, so no TCP connection on this port is established.

 

2. When I access panorama IP on port 3978 I will get window to select which certificate I want use to authenticate myself

AdamHP_0-1646253059854.png

After I click on OK, I see that panorama is using self-signed cert issued to localhost

 

3. In log only entry like this was found with the serial of the FW

2022-03-02 15:22:45.032 +0100 String is <devices>
  <entry name="000000000001"/>
  <entry name="000000000000"/>
</devices>
2022-03-02 15:22:45.032 +0100 After str: <devices>
  <entry name="000000000001"/>
  <entry name="000000000000"/>
</devices>

 

Cyber Elite
Cyber Elite

Thank you for reply @AdamHP

 

If TCP connection is not established, it seems it is failing during initial connection setup. Would it be possible to take packet capture on Panorama side: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CleECAS to see at what phase it is failing and whether any of the side is sending RST?

 

Kind Regards

Pavel

Help the community: Like helpful comments and mark solutions.

Hello Pavel, 

sample output of TCP dump is here(it is looping this same sequence all the time from what I saw in TCP dump)

23:08:44.614213 IP Firewall.54198 > Panorama.pan-panorama: Flags [S], seq 4083635727, win 29200, options [mss 1260,sackOK,TS val 4055216138 ecr 0,nop,wscale 7], length 0
23:08:44.614259 IP Panorama.pan-panorama > Firewall.54198: Flags [S.], seq 4178576975, ack 4083635728, win 24960, options [mss 1260,sackOK,TS val 2636905311 ecr 4055216138,nop,wscale 7], length 0
23:08:44.641051 IP Firewall.54198 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216165 ecr 2636905311], length 0
23:08:44.641107 IP Firewall.54198 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0
23:08:44.993054 IP Firewall.54204 > Panorama.pan-panorama: Flags [S], seq 177465872, win 29200, options [mss 1260,sackOK,TS val 4055216517 ecr 0,nop,wscale 7], length 0
23:08:44.993100 IP Panorama.pan-panorama > Firewall.54204: Flags [S.], seq 1639741662, ack 177465873, win 24960, options [mss 1260,sackOK,TS val 2636905690 ecr 4055216517,nop,wscale 7], length 0
23:08:45.018435 IP Firewall.54204 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216543 ecr 2636905690], length 0
23:08:45.031530 IP Firewall.54204 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0

Cyber Elite
Cyber Elite

Thank you for getting packet capture @AdamHP

 

Based on packet capture, the Firewall is resetting the connection by setting RST flag. Based on all information that you supplied, I am not clear why this is happening. Would it be possible to look into logs on Firewall to see it can provide more details: tail lines 1000 mp-log ms.log

 

Kind Regards

Pavel 

Help the community: Like helpful comments and mark solutions.

L2 Linker

Hi @AdamHP

 

To be double sure that the firewall is the one sending the RST and not any intermediate device, I would take a simultaneous packet capture on Firewall and Panorama. An alternate way would be to compare the TTL value (in the IP Header) of the RST with the SYN packet (if both are sent from the same host, TTL will be the same).

 

Regards

Hi Pavel, 

this look like the interesting part from this log as it is looping there.

2022-03-04 09:35:18.328 +0100 COMM: connection established. sock=29 remote ip=PANORAMA_IP port=3978 local port=35712
2022-03-04 09:35:18.328 +0100 cms agent: Pre. send buffer limit=46080. s=29
2022-03-04 09:35:18.328 +0100 cms agent: Post. send buffer limit=425984. s=29
2022-03-04 09:35:18.328 +0100 Error:  cs_load_certs_ex(cs_common.c:654): keyfile not exists
2022-03-04 09:35:18.328 +0100 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:876): cms agent: cs_load_certs_ex failed
2022-03-04 09:35:18.328 +0100 cmsa: client will use default context
2022-03-04 09:35:18.331 +0100 Error:  sc3_ca_exists(sc3_certs.c:221): SC3: Failed to get the current CA name.
2022-03-04 09:35:18.331 +0100 Warning:  sc3_init_sc3(sc3_utils.c:351): SC3: Failed to get the Current CC name
2022-03-04 09:35:18.331 +0100 SC3: CA: '', CC/CSR: '9469a205-8e13-46ed-879c-13d45a0ae772'
2022-03-04 09:35:18.332 +0100 Warning:  sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI
2022-03-04 09:35:18.332 +0100 Warning:  sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN
2022-03-04 09:35:18.341 +0100 Warning:  sc3_init_sctx(sc3_ctx.c:323): SC3: not set, skip cert loading
2022-03-04 09:35:18.341 +0100 SC3A: using SNI (from AK): 75220d86-f64a-4a64-b542-1b81b8cae893
2022-03-04 09:35:18.341 +0100 SC3A: using sc3 ctx with no cert
2022-03-04 09:35:18.342 +0100 Error:  pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:1196): panorama agent: SSL connect error. sock=29 err=5

It look like it can connect which is the traffic I can see in the Datacenter Firewall and the problem is with SSL? 

Cyber Elite
Cyber Elite

Thank you for reply and getting logs @AdamHP

 

The Error 5 is "SSL verification failure". Unless you configured mutual SSL authentication, only Panorama has to present a certificate: https://docs.paloaltonetworks.com/panorama/10-2/panorama-admin/set-up-panorama/set-up-authentication...

 

Would it be possible one more time to check the certificate on Panorama? The certificate is self signed, however could you confirm: valid to, issuer, subject,...?

 

Kind Regards

Pavel

Help the community: Like helpful comments and mark solutions.

Hi Pavel,

if you mean cert on https port 3978, I am getting this.

AdamHP_0-1646646144612.png

It is issued to my panorama IP address. 

L0 Member

Check under Monitor -> Session Browser under each firewall it flows through, even the source firewall.

Had the same problem and it turns out Panorama traffic won't match a catch all, it must specifically be application Panorama.

L1 Bithead

Thank you all for help, it turns out that on source firewall PA-440 I allow traffic for Panorama Application on its default port 3978, but in firewall monitor I found that the flow is recognized as ssl on port 3978 and this was blocked. 
I was thinking that once there is Panorama app it will be match and didn't check this, but I was wrong. 🙂
I had simmilar issue today on another firewall in Data Center where we had rule for WinRM (microsoft-remote-management), which was working fine before but now after some updates it is recognized as web-browsing on the WinRM port 5985.
So the solution is to not trust the Palo Alto application matching and always check the flows. 

L2 Linker

I have also PA-440 on 10.2 and Panorama on 10.2. And same issue, FW disconnecting. I think this is a bug with 10.2

--
PA-440, PA-1410, Cortex XDR
  • 25066 Views
  • 20 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!