09-20-2023 10:29 AM
- We ran into an issue where the commits from Panorama were failing with error:
• . Performing panorama connectivity check (attempt 1 of 1)
• . Panorama connectivity check failed for xxxx. Reason: TCP channel setup failed, reverting configuration
• . Configuration reverted successfully
- We checked No validation errors while the commit failed
- run show panorama-status on the managed firewall
> show panorama-status
Panorama Server 1 : xxxx
Connected : yes
HA state : Unknown
- Check if Pings between the Firewalls and Panorama are working
> ping host x.x.x.x - 100 success
- Check if Netstat output on the Firewalls show connnections are Established to the Panorama on port 3978.
> show netstat all yes numeric-hosts yes numeric-ports yes
tcp x.x.x.x:3978 ESTABLISHED
- Connectivity looks fine, tried restarting the management server process
> debug software restart process management-server
- We tried to add the ip address instead of the hostname - but it would still fail
- We tried to push the Device-group for the other cluster as well and it fails with the same reason
- We see show system resources – pointing at 3 zombie processes in the panorama
- We see system files are present
- Analyzing the TSF uploaded – we see
2 (2023-09-19 11:07:19) – Critical System(java) - An Out of Memory condition occured and restarted the process
Sep 19 11:04:43 CMS init: Id "s0" respawning too fast: disabled for 5 minutes
Sep 19 11:07:18 CMS klogd: conn12 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Sep 19 11:07:19 CMS klogd: Out of memory: Kill process 3399 (java) score 504 or sacrifice child
Sep 19 11:07:19 CMS klogd: Killed process 3399 (java) total-vm:686690928kB, anon-rss:8422608kB, file-rss:0kB
We also see
2 (2023-09-13 23:31:25)- Critical System (nodejs ) – Process Restarts - nodejs restarted unexpectedly however no known issue was identified, and we see the java core files which explains the restarts.
- Reboot didn't make a change, we still saw the OOM errors
- Lastly also upgraded the Pano to : 10.1.10-h2 but the issue persists
