Huge drop in Panorama syslog forwarding!

[Mohamed_Haneef]

Hello Team,

I have M-500 Panorama appliances in the active-passive HA. Following are my queries;

1- I can see that active and passive panoramas forward logs to Syslog destinations. Sometimes the passive Panorama is forwarding more logs. Is that normal?

2- I can see a huge drop in the Syslog forwarded by both the Panoramas. The below command output shows the drop count, which is much more than the Syslog sent count:

   debug log-collector log-collection-stats show log-forwarding-stats

3- I can only see the detailed traffic logs for 7 days.

Current PAN-OS: 9.1.10

show system disk-space

------------------------------------------

Filesystem Size Used Avail Use% Mounted on
/dev/root 16G 4.2G 11G 29% /
none 63G 112K 63G 1% /dev
/dev/sda5 38G 19G 18G 53% /opt/pancfg
/dev/sda6 23G 2.9G 19G 14% /opt/panrepo
tmpfs 63G 110M 63G 1% /dev/shm
cgroup_root 63G 0 63G 0% /cgroup
/dev/sda8 129G 71G 51G 59% /opt/panlogs
/dev/loop0 32G 177M 30G 1% /opt/logbuffer
/dev/md1 1.8T 1.4T 404G 77% /opt/panlogs/ld1
/dev/md2 1.8T 1.3T 411G 77% /opt/panlogs/ld2

Kindly suggest.

Thanks

[PavelK]

Thank you for posting your questions @Mohamed_Haneef 

1.)

I assume that you are using Panorama with local Log Collector. Even though Panorama has Active and Passive status for management functions, the log collectors process incoming logs from Firewalls regardless of the Active / Passive status. Depending on how you configured Device Log Forwarding Preferences each of the Log Collector can be ingesting different logs, then if you configure Syslog under Collector Log Forwarding, you might be seeing that Log Collector running on Passive Panorama node might actually be sending more logs to 3rd party system than your Active Panorama node. You can verify current log rate status by going to: Panorama > Managed Collectors > Statistics > Performance (Average Logs/sec). If you in the future build dedicated Log Collector, then it will be dedicated Log Collector sending logs to 3rd party system and not Panorama Manager.

2.)

To be honest, I never looked into this, but only for comparison I checked a few Log Collectors and I can also see: "syslog dropped count" is high, but in you case it does not sound healthy. Are you seeing anything unusual in the System Log? You can use filter: ( subtype eq syslog ).

3.)

Could you go to: Panorama > Collector Groups > [Log Collector Name] > Log Storage > Detailed Firewall Logs and check your allocated quota? If you have unallocated space then try to increase it from unallocated space, otherwise try to reduce slightly quota from for example: "Summary Firewall Logs" or "Infrastructure and Audit Logs" and allocated it to: "Detailed Firewall Logs" to see it makes a difference.

Kind Regards

Pavel

[Mohamed_Haneef]

Thanks for your reply. Sorry, I was offline for few days. 

2-There is no syslog related system log. The drop is still there, a support case also opened for this.

3-Inbound log rate in Panorama is quite high. TAC advised to scale up the memory. Also looking into optimizing some policies which are logging even session start.

Many thanks:)