Are logs lost when log discarded (queue full) increases?

hyeongchanlee — Thu, 04 Aug 2022 08:40:20 GMT

Hi everyone

I changed last week from pa-3020 to pa-3220.
However, the log looks abnormal (7-8 minutes delay).
Looking at the log-receiver status with the command below, log discarded (queue full) is continuously increasing.
Does this mean log loss?
How can I solve this?

admin@PA-3220(active)> debug log-receiver statistics

Logging statistics
------------------------------ -----------
Log incoming rate: 223/sec
Log written rate: 800/sec
Corrupted packets: 0
Corrupted URL packets: 0
Corrupted HTTP HDR packets: 0
Corrupted HTTP HDR Insert packets: 0
Corrupted EMAIL HDR packets: 0
Logs discarded (queue full): 429312640 <<< continuously increasing
Traffic logs written: 17568093
GTP logs written: 0
Tunnel logs written: 0
Auth logs written: 0
Config logs written: 1
System logs written: 15306
Alarm logs written: 0
Userid logs written: 1112654
SCTP logs written: 0
GlobalProtect logs written: 0
DECRYPTION logs written: 0
URL logs written: 503413
Wildfire logs written: 12
Anti-virus logs written: 0
Maching Learning-virus logs written: 0
Wildfire Anti-virus logs written: 0
Spyware logs written: 366410
Spyware-DNS logs written: 0
Attack logs written: 0
Vulnerability logs written: 0
Data logs written: 0
Wif logs written: 0
Fileext logs written: 1632
Fileext logs URL not written: 1632
Fileext logs URL not written (timedout): 0
URL cache age out count: 0
URL cache full count: 0
URL cache key exist count: 143
URL cache wrt incomplete http hdrs count: 0
URL cache rcv http hdr before url count: 0
URL cache full drop count(url log not received): 0
URL cache age out drop count(url log not received): 0
Email hdr cache count: 0
Email hdr cache hit count: 0
HTTP hdr insertion received: 0
HTTP hdr insertion processed: 0
HTTP hdr insert no URL drop count: 0
HTTP hdr insert with invalid URL log: 0
HTTP hdr insert with values exceeded max allowed length: 0
Traffic alarms dropped due to sysd write failures: 0
Traffic alarms dropped due to global rate limiting: 0
Traffic alarms dropped due to each source rate limiting: 0
Traffic alarms generated count: 0
Netflow incoming count: 54975992
Log Forward count: 0
Log Forward discarded (queue full) count: 0
Log Forward discarded (send error) count: 0
Total logs not written due to disk unavailability: 0
Logs not written since disk became unavailable: 0
HIP Report logs received: 0

Summary Statistics:
Num current entries in trsum:8544
Num cumulative entries in trsum:9546424
Num current entries in thsum:1018
Num cumulative entries in thsum:869823
Num current entries in urlsum:0
Num cumulative entries in urlsum:0
Num current entries in gtpsum:0
Num cumulative entries in gtpsum:0
Num current entries in sctpsum:0
Num cumulative entries in sctpsum:0
Num current drop entries in trsum:0

Re: Are logs lost when log discarded (queue full) increases?

reaper — Thu, 04 Aug 2022 09:58:36 GMT

this does indeed mean that logs are being discarded (lost)

you could look into decreasing logging on some extremely chatty applications like DNS by creating a rule specific to these applications and disabling logging

topic Re: Are logs lost when log discarded (queue full) increases? in General Topics

Are logs lost when log discarded (queue full) increases?

Re: Are logs lost when log discarded (queue full) increases?