02-28-2022 06:44 AM
Hello I have new deployed Panorama and new PA-440 Firewall.
I setup Panorama with all basic settings like IP address/netmask, default GW, DNS, it has license assigned.
Next I generated AuthKey for the firewalls with validity for 10 days and without SN specified.
PA-440 is in remote location and has a basic WAN setup and IPSec VPN to my datacenter where panorama is.
It has a vlan interface setup in my internal zone and set as source for every service.
I am able to ping Panorama from the PA-440 so network over VPN is working.
When I setup Panorama IP with Auth Key on the firewall and add Firewall on panorama by the Serial Number I still see PA-440 in panorama as Disconnected.
I checked the DataCenter firewall where IPSec is terminated and I can''t see in logs any blocked traffic in between these two.
Port 3978 for Panorama is enabled in security rules and I can see some ssl traffic is passing in Datacenter over this port.
Is there something else I forgott to setup or something else I need to check in order to be able to manage this Firewall by Panorama?
02-28-2022 10:41 AM - edited 02-28-2022 10:45 AM
Can you these given ports as per your connectivity between FWs and Panorama. And same time can you check the logs
in >monitor>loigs>system level
show panorama status
show devices all
verify both commands.
Panorama APP-ID: When the managed firewall communicates with Panorama, by default this traffic is sent over the MGT interface. Because traffic leaving the MGT interface of the firewall is not subject to a Security policy check, no additional Security policy rule configuration is necessary.
However, in some deployments, the administrator might choose to send management traffic through one of the data-plane interfaces of the firewall. In this case, remember to create a Security policy rule to allow the Panorama application. Otherwise, the firewall denies communications with Panorama.
and try to see reachability:
Use ping from the firewall or Panorama command line ping count <integer> source <IP-address> host <IP-address
and try pcap on mgmt using tcpdump
•Run tcpdump from the command line of Panorama or the firewall to capture the traffic. When you have enough data, press Ctrl+C to stop the capture.
Example: tcpdump filter “host 10.1.10.10
03-01-2022 11:58 AM - edited 03-04-2022 01:35 AM
I am using dataplane port instead of Management as the connection to panorama is anyway over the IPSec VPN connection.
The coresponding Firewall rules are applied on both Firewalls and they are passing the traffic on this ports. Also ping works fine.
From Firewall: show panorama-status
Panorama Server 1 : 192.168.1.20 Connected : no HA state : disconnected
From Panorama: show devices all
Serial Hostname IPv4 IPv6 Connected -------------------------------------------------------------------------- 00000000XXXX no Wildfire Real-time Stream Disabled VPN Disable Mode: no Operational Mode: normal Certificate Status: Certificate subject Name: Certificate expiry at: Connected at: Custom certificate Used: Last masterkey push status: Unknown Last masterkey push timestamp: none Express mode: no Device cert present : Device cert expiry date : N/A
I run also a PCAP and I can see on panorama and also on Firewall traffic from the oposite side.
I also check all this from this checklist:
1. IP connectivity is there
2. Port 3978 and also others required for panorama are open
3. DeviceCert on Panorama installed
4. Serial number of device is correct
5. Management profile set up on this interface used for communication
6. Panorama is on version 10.2, Firewall 10.1.4
7. MTU on tunnel interface lowered
8.Time synchronized using NTP
What else I can check?
03-02-2022 04:19 AM
Thank you for supplying additional information @AdamHP
Could you check from Panorama's CLI whether TCP connection is established by: show netstat numeric yes | match 3978 ?
Could you check status of Panorama's certificate from browser: https://<panorama ip>:3978 ?
Could you also check log on Panorama side from CLI: tail lines 500 mp-log configd.log ? Search events corresponding with Firewall's Serial Number.
03-02-2022 12:36 PM
1. netstat is showing no match, so no TCP connection on this port is established.
2. When I access panorama IP on port 3978 I will get window to select which certificate I want use to authenticate myself
After I click on OK, I see that panorama is using self-signed cert issued to localhost
3. In log only entry like this was found with the serial of the FW
2022-03-02 15:22:45.032 +0100 String is <devices> <entry name="000000000001"/> <entry name="000000000000"/> </devices> 2022-03-02 15:22:45.032 +0100 After str: <devices> <entry name="000000000001"/> <entry name="000000000000"/> </devices>
03-02-2022 01:55 PM
Thank you for reply @AdamHP
If TCP connection is not established, it seems it is failing during initial connection setup. Would it be possible to take packet capture on Panorama side: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CleECAS to see at what phase it is failing and whether any of the side is sending RST?
03-02-2022 02:32 PM
sample output of TCP dump is here(it is looping this same sequence all the time from what I saw in TCP dump)
23:08:44.614213 IP Firewall.54198 > Panorama.pan-panorama: Flags [S], seq 4083635727, win 29200, options [mss 1260,sackOK,TS val 4055216138 ecr 0,nop,wscale 7], length 0 23:08:44.614259 IP Panorama.pan-panorama > Firewall.54198: Flags [S.], seq 4178576975, ack 4083635728, win 24960, options [mss 1260,sackOK,TS val 2636905311 ecr 4055216138,nop,wscale 7], length 0 23:08:44.641051 IP Firewall.54198 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216165 ecr 2636905311], length 0 23:08:44.641107 IP Firewall.54198 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0 23:08:44.993054 IP Firewall.54204 > Panorama.pan-panorama: Flags [S], seq 177465872, win 29200, options [mss 1260,sackOK,TS val 4055216517 ecr 0,nop,wscale 7], length 0 23:08:44.993100 IP Panorama.pan-panorama > Firewall.54204: Flags [S.], seq 1639741662, ack 177465873, win 24960, options [mss 1260,sackOK,TS val 2636905690 ecr 4055216517,nop,wscale 7], length 0 23:08:45.018435 IP Firewall.54204 > Panorama.pan-panorama: Flags [.], ack 1, win 229, options [nop,nop,TS val 4055216543 ecr 2636905690], length 0 23:08:45.031530 IP Firewall.54204 > Panorama.pan-panorama: Flags [R.], seq 1, ack 1, win 229, length 0
03-03-2022 04:57 PM
Thank you for getting packet capture @AdamHP
Based on packet capture, the Firewall is resetting the connection by setting RST flag. Based on all information that you supplied, I am not clear why this is happening. Would it be possible to look into logs on Firewall to see it can provide more details: tail lines 1000 mp-log ms.log
03-03-2022 07:42 PM
To be double sure that the firewall is the one sending the RST and not any intermediate device, I would take a simultaneous packet capture on Firewall and Panorama. An alternate way would be to compare the TTL value (in the IP Header) of the RST with the SYN packet (if both are sent from the same host, TTL will be the same).
03-04-2022 01:40 AM
this look like the interesting part from this log as it is looping there.
2022-03-04 09:35:18.328 +0100 COMM: connection established. sock=29 remote ip=PANORAMA_IP port=3978 local port=35712 2022-03-04 09:35:18.328 +0100 cms agent: Pre. send buffer limit=46080. s=29 2022-03-04 09:35:18.328 +0100 cms agent: Post. send buffer limit=425984. s=29 2022-03-04 09:35:18.328 +0100 Error: cs_load_certs_ex(cs_common.c:654): keyfile not exists 2022-03-04 09:35:18.328 +0100 Error: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:876): cms agent: cs_load_certs_ex failed 2022-03-04 09:35:18.328 +0100 cmsa: client will use default context 2022-03-04 09:35:18.331 +0100 Error: sc3_ca_exists(sc3_certs.c:221): SC3: Failed to get the current CA name. 2022-03-04 09:35:18.331 +0100 Warning: sc3_init_sc3(sc3_utils.c:351): SC3: Failed to get the Current CC name 2022-03-04 09:35:18.331 +0100 SC3: CA: '', CC/CSR: '9469a205-8e13-46ed-879c-13d45a0ae772' 2022-03-04 09:35:18.332 +0100 Warning: sc3_get_current_sc3(sc3_utils.c:179): SC3: failed to get SNI 2022-03-04 09:35:18.332 +0100 Warning: sc3_get_current_sc3(sc3_utils.c:182): SC3: failed to get CCN 2022-03-04 09:35:18.341 +0100 Warning: sc3_init_sctx(sc3_ctx.c:323): SC3: not set, skip cert loading 2022-03-04 09:35:18.341 +0100 SC3A: using SNI (from AK): 75220d86-f64a-4a64-b542-1b81b8cae893 2022-03-04 09:35:18.341 +0100 SC3A: using sc3 ctx with no cert 2022-03-04 09:35:18.342 +0100 Error: pan_cmsa_tcp_channel_setup(src_panos/cms_agent.c:1196): panorama agent: SSL connect error. sock=29 err=5
It look like it can connect which is the traffic I can see in the Datacenter Firewall and the problem is with SSL?
03-05-2022 02:36 PM
Thank you for reply and getting logs @AdamHP
The Error 5 is "SSL verification failure". Unless you configured mutual SSL authentication, only Panorama has to present a certificate: https://docs.paloaltonetworks.com/panorama/10-2/panorama-admin/set-up-panorama/set-up-authentication...
Would it be possible one more time to check the certificate on Panorama? The certificate is self signed, however could you confirm: valid to, issuer, subject,...?
03-07-2022 01:43 AM
if you mean cert on https port 3978, I am getting this.
It is issued to my panorama IP address.
03-10-2022 07:52 AM
Check under Monitor -> Session Browser under each firewall it flows through, even the source firewall.
Had the same problem and it turns out Panorama traffic won't match a catch all, it must specifically be application Panorama.
03-11-2022 08:31 AM
Thank you all for help, it turns out that on source firewall PA-440 I allow traffic for Panorama Application on its default port 3978, but in firewall monitor I found that the flow is recognized as ssl on port 3978 and this was blocked.
I was thinking that once there is Panorama app it will be match and didn't check this, but I was wrong. 🙂
I had simmilar issue today on another firewall in Data Center where we had rule for WinRM (microsoft-remote-management), which was working fine before but now after some updates it is recognized as web-browsing on the WinRM port 5985.
So the solution is to not trust the Palo Alto application matching and always check the flows.
03-30-2022 03:09 PM
I have also PA-440 on 10.2 and Panorama on 10.2. And same issue, FW disconnecting. I think this is a bug with 10.2
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!