Interpreting debug log-receiver statistics command output

[tamilvanan]

Hi All,

We are having issue with management plane CPU going high. Upon checking we had identified the Logrcvr process is consuming more memory during the issue time. 

We are having syslog forwarding profile and Net flow profile configured on the firewall.

Had run the command debug log-receiver statistics and got the below output. Can any please help me out on what the following parameters in the output means : Num cumulative drop entries in trsum, Enqueue Count, Send Count, Netflow incoming count

Log Output:

Logging statistics
------------------------------ -----------
Log incoming rate: 1657/sec
Log written rate: 1657/sec
Corrupted packets: 0
Corrupted URL packets: 0
Corrupted HTTP HDR packets: 0
Corrupted HTTP HDR Insert packets: 0
Corrupted EMAIL HDR packets: 0
Logs discarded (queue full): 0
Traffic logs written: 3891600225
GTP logs written: 0
Tunnel logs written: 0
Auth logs written: 0
Userid logs written: 19069
SCTP logs written: 0
GlobalProtect logs written: 182511
DECRYPTION logs written: 11886
URL logs written: 0
Wildfire logs written: 96605
Anti-virus logs written: 199
Maching Learning-virus logs written: 0
lines 1-23 Wildfire Anti-virus logs written: 1768
Spyware logs written: 49341679
Spyware-DNS logs written: 20973
Attack logs written: 0
Vulnerability logs written: 46438318
Data logs written: 0
Wif logs written: 0
Fileext logs written: 0
Fileext logs URL not written: 0
Fileext logs URL not written (timedout): 0
URL cache age out count: 0
URL cache full count: 0
URL cache key exist count: 0
URL cache wrt incomplete http hdrs count: 0
URL cache rcv http hdr before url count: 0
URL cache full drop count(url log not received): 0
URL cache age out drop count(url log not received): 0
Email hdr cache count: 1695
Email hdr cache hit count: 1970
HTTP hdr insertion received: 0
HTTP hdr insertion processed: 0
HTTP hdr insert no URL drop count: 0
HTTP hdr insert with invalid URL log: 0
lines 24-46 HTTP hdr insert with values exceeded max allowed length: 0
Traffic alarms dropped due to sysd write failures: 0
Traffic alarms dropped due to global rate limiting: 0
Traffic alarms dropped due to each source rate limiting: 0
Traffic alarms generated count: 0
Netflow incoming count: 4053054063
Log Forward count: 14315436
Log Forward discarded (queue full) count: 0
Log Forward discarded (send error) count: 0
Total logs not written due to disk unavailability: 0
Logs not written since disk became unavailable: 0
DPI logs received: 0
HIP Report logs received: 0

Summary Statistics:
Num current entries in trsum:283974
Num cumulative entries in trsum:2399889465
Num current entries in thsum:282
Num cumulative entries in thsum:98934187
Num current entries in urlsum:0
Num cumulative entries in urlsum:0
Num current entries in gtpsum:0
Num cumulative entries in gtpsum:0
lines 47-69 Num current entries in sctpsum:0
Num cumulative entries in sctpsum:0
Num current drop entries in trsum:0
Num cumulative drop entries in trsum:5273142
Num current drop entries in thsum:0
Num cumulative drop entries in thsum:0
Num current drop entries in urlsum:0
Num cumulative drop entries in urlsum:0
Num current drop entries in gtpsum:0
Num cumulative drop entries in gtpsum:0
Num current drop entries in sctpsum:0
Num cumulative drop entries in sctpsum:0
Num current drop entries in desum:0
Num cumulative drop entries in desum:0

External Forwarding stats:
Type   Enqueue Count   Send Count    Drop   Count Queue Depth    Send Rate(last 1min)
syslog 1626503687         1626503687      0         0                                   25922
snmp 0 0 0 0  0
email 0 0 0 0 lines 70-89  0
raw 0 0 0 0  0
http 0 0 0 0  0
autotag 0 0 0 0  0
quarantine 0 0 0 0  0

Re: Log forwarding - Local on Gateway or Panorama 

[reaper · Accepted solution]

Num cumulative drop entries in trsum, total number of "drop" logs in the traffic summary log

Enqueue Count, logs received to be forwarded

Send Count, logs forwarded

Netflow incoming count, received netflow logs

What "size" firewall do you have, the log volume may be reaching the maximum rate the chassis/vm supports causing management slowness

[tamilvanan]

Hi @reaper thanks for the reply.

We are having PA-850 firewall and the management plane is reaching till 80%.

Upon checking the process running the firewall during the issue period the Logrcvr process is consuming more virtual memory. We are having syslog forwarding for every security rule and also have Netflow configured for two interfaces on the firewall

Also checked the Logrcvr log file and could see the traffic and threat logs being flushed constanly. 

Just need to know what are the activities handled by the logrcvr process to minimize the load on the firewall

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
(B 4344 root 20 0 77100 9268 6864 R 100.0 0.2 190806:36 pan_task

(B 4345 root 20 0 77100 10320 7004 R 100.0 0.3 190812:41 pan_task 
(B 4346 root 20 0 72832 9340 6976 R 100.0 0.2 190822:03 pan_task 
(B 4347 root 20 0 76764 9636 7040 R 100.0 0.2 190814:29 pan_task 
(B 4343 root 20 0 102056 26028 6928 R 100.0 0.6 190813:41 pan_task 
(B 6207 root 20 0 2342984 285388 8992 S 50.0 6.9 17861:20 logrcvr