topic Re: System logs stalling same time every day in General Topics

System logs stalling same time every day

PatScott — Thu, 08 Jul 2021 14:11:32 GMT

On our PA3050 the system logs stall each day at 04:01 and then starts again at 20:00

I have verified this happens in both GUI and CLI

No scheduled jobs correlate with the timing of the logs stopping then starting

I have checked show system logdb-quota and everything looks fine

I have checked show system disk-space and we're fine on disk space

I have tried restarting log receiver process with no change - debug software restart process log-receiver

We can see other logged data (Traffic, URL, Unified, Threat )

I have a case open with PA support, but no suggestions I haven't already tried.

I haven't yet tried restarting management server process or tried clearing system logs in the GUI.

Should I try restarting management server process? Since this is only affecting system logs I wasn't sure if that was the way to go. Also not sure how much time I should expect services to be unavailable while this is happening; and also don't want to do anything that might affect our other (traffic, URL, Threat, Unified) logs as this is a stand alone prod appliance.

Also not sure if I should try clearing system logs in Device -> Log settings -> clear system logs would that erase all system logs forever?

Any suggestions welcome.

Re: System logs stalling same time every day

BPry — Thu, 08 Jul 2021 18:56:12 GMT

@PatScott,

Restarting the management server doesn't take very long and I've never noticed missing logs after doing so. The restart would only effect management traffic, it doesn't effect production traffic processing at all.

Also not sure if I should try clearing system logs in Device -> Log settings -> clear system logs would that erase all system logs forever?

Yes, so if you aren't sending those somewhere else they're gone.

For a situation like this I always recommend starting with restarting processes and following it up with an actual device restart if that doesn't work to see if that fixes the issue. I'd get a maintenance window opened and give that a shot.

Re: System logs stalling same time every day

PatScott — Thu, 08 Jul 2021 19:06:07 GMT

@BPry thanks for verifying that. I will start by restarting the management server process. As I understand it, that only affects ability to manage but doesn't interrupt data processing, so I can do that early AM.

Just wanted to verify that was the case.

This one really has me stumped, so appreciate getting help from experienced folks.

If we need to restart the actual device I can get a maintenance window.

Thanks for the reply.

Re: System logs stalling same time every day

PatScott — Fri, 09 Jul 2021 11:24:26 GMT

So that worked so far - debug software restart process management-server
I can now see system logs again. Hopefully they don't stall again.

Thanks again