topic Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures." in General Topics

Log "Number of hints on disk has exceeded 5000 due to log forward failures."

BigPalo — Fri, 26 Oct 2018 12:36:02 GMT

Hi,

We are receiving these logs. We would like to know what is causing these logs and how to solve it.

Thanks 🙂

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

BigPalo — Mon, 29 Oct 2018 12:00:40 GMT

any idea?

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

LukeBullimore — Mon, 29 Oct 2018 16:25:15 GMT

Hey @BigPalo

Check out the below thread, it seems people have resolved the issue by running the command "debug software restart process log-receiver"

https://live.paloaltonetworks.com/t5/General-Topics/General-PA-5220/m-p/192473#M57806

As for the root cause, are you running Panorama?

Cheers,

Luke.

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

BigPalo — Tue, 30 Oct 2018 11:28:23 GMT

Yes, we are running Panorama

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

LukeBullimore — Tue, 30 Oct 2018 13:22:08 GMT

Hey @BigPalo

Cheers for confirming that. Did you restart the log receiver service and did it resolve the issue?

From what I gather, this problem is caused by the send queue being filled up when attempting to forward logs to Panorama. This can be verified by looking at the netstat output "show netstat" and looking at the "Send Queue" column for a socket open on port 10000.

In Panorama, there are a few best practices that we can look at:

1. Has a log forwarding preference list been configured? Panorama -> Collector Groups -> Device Log Forwarding

2. Is "enable redundancy across log collectors" checked?

3. Is "Forward to all collectors in the preference list" checked?

If options two and three are enabled, without the use of the preference list, then all logs will just be sent to one LC, and this will then be copying the logs to the other LCs anyways - causing a lot of stress. At this point the Panorama will start to throttle logs and this is when you will notice the netstat queues increasing.

Cheers,

Luke.

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

JW6224 — Tue, 30 Oct 2018 13:14:24 GMT

In Panorama, there are a few best practices that we can look at:

1. Has a log forwarding preference list been configured? Panorama -> Managed Collectors -> Device Log Forwarding

@LukeBullimore- I think that setting is under the Collector Groups, not Managed Collectors

Good best practices list - much appreciated!

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

LukeBullimore — Tue, 30 Oct 2018 13:21:35 GMT

Hey @JW6224

Whoops yeah that was a typo, I'll correct it now.

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Tue, 30 Oct 2018 15:00:06 GMT

I am still getting this error i ran the command debug restart log receiver

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Tue, 30 Oct 2018 15:24:50 GMT

I see PA is conected to Panorama and we have dedicated log collectors

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

BigPalo — Fri, 02 Nov 2018 11:06:26 GMT

are you still having this issue???

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Fri, 02 Nov 2018 14:58:24 GMT

no.

restarting the log receiver from the root fixed the issue

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

DaxVC — Mon, 18 Feb 2019 08:49:17 GMT

Hi,

What do you exactly mean by restarting the service from the root ?

I'm experiencing the same issue but restarting the service did not resolve the issue on my device ( which is even a standby device )

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

BigPalo — Mon, 11 Mar 2019 11:18:36 GMT

Do you have any news for this issue? we are still having the problem.

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

DaxVC — Mon, 11 Mar 2019 12:33:09 GMT

Hi,

The error dissapeared after restarting the mgmtsrv service and waiting for 8 hours.

We didn't notice the error anymore since then