Panorama - logs displayed for some firewalls are almost 20 hours old

[ChristopherMarston]

Panorama 9.1.2 VM

Both firewalls running 9.1.2 and forwarding logs to Panorama.  For one firewall the logs in Panorama are current.  For the other they are over 19 hours old for the traffic logs.  There is a third firewall where Panorama shows logs more recent than 19 hours but still not current.  NTP settings match on all devices and clocks show the same time.  Panorama CPU is at 1%.  Tried a number of things from KB articles and eventually rebooted the Panorama with no change.  Latency between firewall and Panorama is about 90ms.

From the device with 19 hour old logs in Panorama.

(active)> show logging-status

-----------------------------------------------------------------------------------------------------------------------------
Type Last Log Created Last Log Fwded Last Seq Num Fwded Last Seq Num Acked Total Logs Fwded
-----------------------------------------------------------------------------------------------------------------------------
> CMS 0
Not Sending to CMS 0
> CMS 1
Not Sending to CMS 1

>Log Collector
'Log Collection log forwarding agent' is active and connected to X.X.X.X

config Not Available Not Available 0 171 0
system 2020/10/16 18:42:40 2020/10/17 19:10:38 58072 58062 2377
threat 2020/10/17 03:18:14 2020/10/17 19:10:38 243 186 57
traffic 2020/10/17 00:15:42 2020/10/17 19:46:59 251946749 251939639 17300976
hipmatch Not Available Not Available 0 0 0
gtp-tunnel Not Available Not Available 0 0 0
userid Not Available Not Available 0 0 0
iptag Not Available Not Available 0 0 0
auth Not Available Not Available 0 0 0
sctp Not Available Not Available 0 0 0
globalprotect Not Available Not Available 0 0 0

[MP18]

@ChristopherMarston 

Are all firewalls and Panorama at same physical location?

Does this work perfectly before?

90ms is too high latency.

If you have already rebooted the Panorama then as next step I will restart the management plane of the firewall which does not show

recent logs in Panorama.

Regards

[BPry]

@ChristopherMarston,

Your latency should be perfectly fine between Panorama and the firewalls, that wouldn't cause this large of a delay. If this just started happening on the two non-functioning firewalls I would definitely restart the management server as @MP18 mentioned to make sure that all of the processes are in a good state.

You might want to think about upgrading to 9.1.4 which is the current recommended release. There's a number of addressed Panorama related issues in 9.1.3 and 9.1.4. 

[ChristopherMarston]

The Panorama and firewalls are on two different continents.  The three firewalls were effectively rebooted on Friday due to a maintenance on the ESX server the primary nodes were running on.  The issue does predate that maintenance and I notice today the delay is now over 21 hours.  Thanks for the reply

[ChristopherMarston]

To add...   We are in the process of testing 9.1.4 for deployment.  I don't find anything related in fixed issue for 9.1.4 but I do find PAN-143809 addressed in 9.1.5 but not clear if it's the issue I'm having.  More info for consideration.

Firewall 1 - generating 2 logs p/sec.  No issues in Panorama

Firewall 2 - generating 230 logs p/sec. Over 21 hours delay in traffic logs, threat logs current

Firewall 3 - generating 2897 logs p/sec.  Only 2 hours delay in traffic logs, 3 hours delay for threat logs