topic Re: My PA-1410 logs for single day, why? how to solve? in General Topics

My PA-1410 logs for single day, why? how to solve?

MRamadanAHafiez — Fri, 20 Sep 2024 19:11:48 GMT

Hello Team,

My new PA-1410 logging is not more than a single day when checking the traffic logs.

Previously I had PA-3220 I could checked months of logs.

whats wrong here in the PA-1410 loggin settings?

manager@PA-1410-Main(active)> show system logdb-quota

Quotas:

system: 4.00%, 0.726 GB Expiration-period: 0 days

config: 4.00%, 0.726 GB Expiration-period: 0 days

alarm: 3.00%, 0.544 GB Expiration-period: 0 days

appstat: 4.00%, 0.726 GB Expiration-period: 0 days

hip-reports: 1.00%, 0.181 GB Expiration-period: 0 days

traffic: 29.00%, 5.263 GB Expiration-period: 0 days

threat: 15.00%, 2.722 GB Expiration-period: 0 days

trsum: 7.00%, 1.270 GB Expiration-period: 0 days

hourlytrsum: 3.00%, 0.544 GB Expiration-period: 0 days

dailytrsum: 1.00%, 0.181 GB Expiration-period: 0 days

weeklytrsum: 1.00%, 0.181 GB Expiration-period: 0 days

urlsum: 2.00%, 0.363 GB Expiration-period: 0 days

hourlyurlsum: 1.00%, 0.181 GB Expiration-period: 0 days

dailyurlsum: 1.00%, 0.181 GB Expiration-period: 0 days

weeklyurlsum: 0.75%, 0.136 GB Expiration-period: 0 days

thsum: 2.00%, 0.363 GB Expiration-period: 0 days

hourlythsum: 1.00%, 0.181 GB Expiration-period: 0 days

dailythsum: 1.00%, 0.181 GB Expiration-period: 0 days

weeklythsum: 1.00%, 0.181 GB Expiration-period: 0 days

userid: 1.00%, 0.181 GB Expiration-period: 0 days

iptag: 1.00%, 0.181 GB Expiration-period: 0 days

application-pcaps: 1.00%, 0.181 GB Expiration-period: 0 days

extpcap: 1.00%, 0.181 GB Expiration-period: 0 days

debug-filter-pcaps: 1.00%, 0.181 GB Expiration-period: 0 days

dlp-logs: 1.00%, 0.181 GB Expiration-period: 0 days

hipmatch: 3.00%, 0.544 GB Expiration-period: 0 days

gtp: 2.00%, 0.363 GB Expiration-period: 0 days

gtpsum: 1.00%, 0.181 GB Expiration-period: 0 days

hourlygtpsum: 0.75%, 0.136 GB Expiration-period: 0 days

dailygtpsum: 0.75%, 0.136 GB Expiration-period: 0 days

weeklygtpsum: 0.75%, 0.136 GB Expiration-period: 0 days

auth: 1.00%, 0.181 GB Expiration-period: 0 days

sctp: 0.00%, 0.000 GB Expiration-period: 0 days

sctpsum: 0.00%, 0.000 GB Expiration-period: 0 days

hourlysctpsum: 0.00%, 0.000 GB Expiration-period: 0 days

dailysctpsum: 0.00%, 0.000 GB Expiration-period: 0 days

weeklysctpsum: 0.00%, 0.000 GB Expiration-period: 0 days

decryption: 1.00%, 0.181 GB Expiration-period: 0 days

desum: 1.00%, 0.181 GB Expiration-period: 0 days

hourlydesum: 0.00%, 0.000 GB Expiration-period: 0 days

dailydesum: 0.00%, 0.000 GB Expiration-period: 0 days

weeklydesum: 0.00%, 0.000 GB Expiration-period: 0 days

globalprotect: 1.00%, 0.181 GB Expiration-period: 0 days

Disk usage:

traffic: Logs and Indexes: 4.3G Current Retention: 1 days

threat: Logs and Indexes: 2.0G Current Retention: 2 days

system: Logs and Indexes: 254M Current Retention: 35 days

config: Logs and Indexes: 132M Current Retention: 35 days

alarm: Logs and Indexes: 13M Current Retention: 35 days

trsum: Logs and Indexes: 1.4G Current Retention: 2 days

hourlytrsum: Logs and Indexes: 8.0K Current Retention: 0 days

dailytrsum: Logs and Indexes: 197M Current Retention: 2 days

weeklytrsum: Logs and Indexes: 200M Current Retention: 12 days

thsum: Logs and Indexes: 163M Current Retention: 35 days

hourlythsum: Logs and Indexes: 178M Current Retention: 27 days

dailythsum: Logs and Indexes: 137M Current Retention: 27 days

weeklythsum: Logs and Indexes: 32M Current Retention: 26 days

appstatdb: Logs and Indexes: 234M Current Retention: 35 days

userid: Logs and Indexes: 147M Current Retention: 2 days

iptag: Logs and Indexes: 57M Current Retention: 16 days

hipmatch: Logs and Indexes: 126M Current Retention: 27 days

hip-reports: Logs and Indexes: Current Retention: 0 days

extpcap: Logs and Indexes: 98M Current Retention: 35 days

urlsum: Logs and Indexes: 383M Current Retention: 1 days

hourlyurlsum: Logs and Indexes: 195M Current Retention: 0 days

dailyurlsum: Logs and Indexes: 187M Current Retention: 2 days

weeklyurlsum: Logs and Indexes: 164M Current Retention: 12 days

gtp: Logs and Indexes: 13M Current Retention: 35 days

gtpsum: Logs and Indexes: 13M Current Retention: 35 days

hourlygtpsum: Logs and Indexes: 296K Current Retention: 0 days

dailygtpsum: Logs and Indexes: 288K Current Retention: 0 days

weeklygtpsum: Logs and Indexes: 48K Current Retention: 0 days

auth: Logs and Indexes: 113M Current Retention: 35 days

sctp: Logs and Indexes: 13M Current Retention: 35 days

sctpsum: Logs and Indexes: 13M Current Retention: 35 days

hourlysctpsum: Logs and Indexes: 8.0K Current Retention: 0 days

dailysctpsum: Logs and Indexes: 8.0K Current Retention: 0 days

weeklysctpsum: Logs and Indexes: 8.0K Current Retention: 0 days

decryption: Logs and Indexes: 148M Current Retention: 2 days

desum: Logs and Indexes: 191M Current Retention: 21 days

hourlydesum: Logs and Indexes: 8.0K Current Retention: 0 days

dailydesum: Logs and Indexes: 8.0K Current Retention: 0 days

weeklydesum: Logs and Indexes: 8.0K Current Retention: 0 days

globalprotect: Logs and Indexes: 139M Current Retention: 35 days

application: Logs and Indexes: 161M Current Retention: 24 days

filters: Logs and Indexes: 4.0K Current Retention: 0 days

dlp: Logs and Indexes: 4.0K Current Retention: 0 days

hip_report_base: Logs and Indexes: 2.7M Current Retention: N/A

wildfire: Logs and Indexes: 48K Current Retention: N/A

Space reserved for cores: 0MB

Re: My PA-1410 logs for single day, why? how to solve?

BPry — Sat, 21 Sep 2024 03:54:48 GMT

@MRamadanAHafiez,

So good news and bad news. The good news is that nothing is wrong with your PA-1410 and it's functioning as designed and from what you've shared you've successfully allocated the entire disk available for logs. The bad news is that nothing is wrong and there's nothing you can readily do to just fix this.

Your PA-1410 has a much smaller disk than what your PA-3220 did, which is just a consequence of moving down in chassis. The PA-3220 has a 240GB SSD where your PA-1410 only has 128GB available. That translates by default to your PA-1410 having 30GB available for logs while your PA-3220 had over quadruple that at 132GB.

The size difference doesn't account for the drop from months to a single day however, but your traffic log size that you have indicated is essentially not going to allow for much more than a days worth of logs. With that large of a difference I would be trying to see if you're logging something that you maybe weren't previously that is extremely chatty, for example DNS traffic if you weren't logging that previously.

Since simply replacing hardware is a poor replacement to a logging retention issue, your cheapest path forward is going to be setting up a SIEM for long-term log storage. Something like Graylog can be had for free as long as you have a system to run it and the storage to actually store the logs long-term. I'd personally try adjusting some of your allocation to get a longer traffic log retention on the PA-1410 itself just because that makes troubleshooting a bit easier, but ultimately you'll have to shift your long-term logs somewhere and get used to searching things a bit differently.

Re: My PA-1410 logs for single day, why? how to solve?

MRamadanAHafiez — Sat, 21 Sep 2024 06:28:42 GMT

Thanx Bro 😞 I felt like I got stuck in this small box.

About SIEM, I already have deployed elastics with palaoto logs forwarded to it. and according to you, some logging in the policies must be done then.

By the way, one last ques. Do you recommed to set the log to "Log ar session start, or log at session end" ? as long as we won;t be able to set to both in most of the policies.

Thanks again.

Re: My PA-1410 logs for single day, why? how to solve?

BPry — Sat, 21 Sep 2024 06:44:01 GMT

@MRamadanAHafiez,

Default to log at session end and evaluate the importance of having the immediate knowledge of a session that is on-going without looking at the session table. The reason only logging at the end of the session is the default is that the session table is an easy way to look at all on-going sessions without looking at the log files themselves. You never really need logs available solely at the start of the session because that session itself is visible directly in the session table.

There's limited policies that I will have log-start and log-end enabled, and it's usually just traditionally long-running sessions that I care to track via automation. For example SSH/RDP/SMB are all connections that can be open for a real long time if they are actively passing traffic, but I don't want to have a process to look at closed sessions and on-going sessions because I'd have to develop that automation around two different sets of data. I personally just find it easier to enable log-start and have my automation generate alerts based on the traffic logs instead of both the logs and the session table to keep things a bit cleaner. If I was running into issues with log retention then I would personally likely take the time to adjust my automation to actively look at the session table in addition to just the logs themselves.

I don't see too much use in generating logs at session start unless I'm actively monitoring those connections. If someone asks you to troubleshoot something that is on-going it's as easy as going into the session table instead of the traffic logs to see how the traffic is being processed. You can adjust alerting to account for not having logs at session start, you can adjust automation to look at the session table instead of the logs, and just in general there's few session that I feel the need to know immediately as soon as the session kicks off that isn't satisfied by the session table.

Re: My PA-1410 logs for single day, why? how to solve?

JimCox — Mon, 30 Dec 2024 15:50:38 GMT

That does nto explain why his retention period for Traffic is set to 1 day, it should be zero, the default and the log should roll over as it fills.