Panorama - logs displayed for some firewalls are almost 20 hours old

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Reply
Highlighted

Panorama - logs displayed for some firewalls are almost 20 hours old

Panorama 9.1.2 VM

 

Both firewalls running 9.1.2 and forwarding logs to Panorama.  For one firewall the logs in Panorama are current.  For the other they are over 19 hours old for the traffic logs.  There is a third firewall where Panorama shows logs more recent than 19 hours but still not current.  NTP settings match on all devices and clocks show the same time.  Panorama CPU is at 1%.  Tried a number of things from KB articles and eventually rebooted the Panorama with no change.  Latency between firewall and Panorama is about 90ms.

 

From the device with 19 hour old logs in Panorama.

(active)> show logging-status

-----------------------------------------------------------------------------------------------------------------------------
Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded
-----------------------------------------------------------------------------------------------------------------------------
> CMS 0
Not Sending to CMS 0
> CMS 1
Not Sending to CMS 1

>Log Collector
'Log Collection log forwarding agent' is active and connected to X.X.X.X


config Not Available Not Available 0 171 0
system 2020/10/16 18:42:40 2020/10/17 19:10:38 58072 58062 2377
threat 2020/10/17 03:18:14 2020/10/17 19:10:38 243 186 57
traffic 2020/10/17 00:15:42 2020/10/17 19:46:59 251946749 251939639 17300976
hipmatch Not Available Not Available 0 0 0
gtp-tunnel Not Available Not Available 0 0 0
userid Not Available Not Available 0 0 0
iptag Not Available Not Available 0 0 0
auth Not Available Not Available 0 0 0
sctp Not Available Not Available 0 0 0
globalprotect Not Available Not Available 0 0 0

Highlighted
Cyber Elite

@ChristopherMarston 

 

Are all firewalls and Panorama at same physical location?

Does this work perfectly before?

90ms is too high latency.

 

If you have already rebooted the Panorama then as next step I will restart the management plane of the firewall which does not show

recent logs in Panorama.

 

Regards

 

 

MP
Highlighted
Cyber Elite

@ChristopherMarston,

Your latency should be perfectly fine between Panorama and the firewalls, that wouldn't cause this large of a delay. If this just started happening on the two non-functioning firewalls I would definitely restart the management server as @MP18 mentioned to make sure that all of the processes are in a good state.

You might want to think about upgrading to 9.1.4 which is the current recommended release. There's a number of addressed Panorama related issues in 9.1.3 and 9.1.4. 

Highlighted

The Panorama and firewalls are on two different continents.  The three firewalls were effectively rebooted on Friday due to a maintenance on the ESX server the primary nodes were running on.  The issue does predate that maintenance and I notice today the delay is now over 21 hours.  Thanks for the reply

Highlighted

To add...   We are in the process of testing 9.1.4 for deployment.  I don't find anything related in fixed issue for 9.1.4 but I do find PAN-143809 addressed in 9.1.5 but not clear if it's the issue I'm having.  More info for consideration.

Firewall 1 - generating 2 logs p/sec.  No issues in Panorama

Firewall 2 - generating 230 logs p/sec. Over 21 hours delay in traffic logs, threat logs current

Firewall 3 - generating 2897 logs p/sec.  Only 2 hours delay in traffic logs, 3 hours delay for threat logs

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!