- We ran into an issue where the commits from Panorama were failing with error:
• . Performing panorama connectivity check (attempt 1 of 1)
• . Panorama connectivity check failed for xxxx. Reason: TCP channel setup failed, reverting configuration
• . Configuration reverted successfully
- We checked No validation errors while the commit failed
- run show panorama-status on the managed firewall
> show panorama-status
Panorama Server 1 : xxxx
Connected : yes
HA state : Unknown
- Check if Pings between the Firewalls and Panorama are working
> ping host x.x.x.x - 100 success
- Check if Netstat output on the Firewalls show connnections are Established to the Panorama on port 3978.
> show netstat all yes numeric-hosts yes numeric-ports yes
tcp x.x.x.x:3978 ESTABLISHED
- Connectivity looks fine, tried restarting the management server process
> debug software restart process management-server
- We tried to add the ip address instead of the hostname - but it would still fail
- We tried to push the Device-group for the other cluster as well and it fails with the same reason
- We see show system resources – pointing at 3 zombie processes in the panorama
- We see system files are present
- Analyzing the TSF uploaded – we see
2 (2023-09-19 11:07:19) – Critical System(java) - An Out of Memory condition occured and restarted the process
Sep 19 11:04:43 CMS init: Id "s0" respawning too fast: disabled for 5 minutes
Sep 19 11:07:18 CMS klogd: conn12 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Sep 19 11:07:19 CMS klogd: Out of memory: Kill process 3399 (java) score 504 or sacrifice child
Sep 19 11:07:19 CMS klogd: Killed process 3399 (java) total-vm:686690928kB, anon-rss:8422608kB, file-rss:0kB
We also see
2 (2023-09-13 23:31:25)- Critical System (nodejs ) – Process Restarts - nodejs restarted unexpectedly however no known issue was identified, and we see the java core files which explains the restarts.
- Reboot didn't make a change, we still saw the OOM errors
- Lastly also upgraded the Pano to : 10.1.10-h2 but the issue persists
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!