Log "Number of hints on disk has exceeded 5000 due to log forward failures."

[BigPalo]

Hi,

We are receiving these logs. We would like to know what is causing these logs and how to solve it.

Thanks 🙂

[BigPalo]

any idea?

[LukeBullimore]

Hey @BigPalo

Check out the below thread, it seems people have resolved the issue by running the command "debug software restart process log-receiver"

https://live.paloaltonetworks.com/t5/General-Topics/General-PA-5220/m-p/192473#M57806

As for the root cause, are you running Panorama?

Cheers,

Luke.

[BigPalo]

Yes, we are running Panorama

[LukeBullimore]

Hey @BigPalo

Cheers for confirming that. Did you restart the log receiver service and did it resolve the issue?

From what I gather, this problem is caused by the send queue being filled up when attempting to forward logs to Panorama. This can be verified by looking at the netstat output "show netstat" and looking at the "Send Queue" column for a socket open on port 10000.

In Panorama, there are a few best practices that we can look at:

1. Has a log forwarding preference list been configured? Panorama -> Collector Groups -> Device Log Forwarding

2. Is "enable redundancy across log collectors" checked?

3. Is "Forward to all collectors in the preference list" checked?

If options two and three are enabled, without the use of the preference list, then all logs will just be sent to one LC, and this will then be copying the logs to the other LCs anyways - causing a lot of stress. At this point the Panorama will start to throttle logs and this is when you will notice the netstat queues increasing.

Cheers,

Luke.

[JW6224]

---

In Panorama, there are a few best practices that we can look at:

1. Has a log forwarding preference list been configured? Panorama -> Managed Collectors -> Device Log Forwarding

---

@LukeBullimore- I think that setting is under the Collector Groups, not Managed Collectors

Good best practices list - much appreciated!

[LukeBullimore]

Hey @JW6224

Whoops yeah that was a typo, I'll correct it now. 

[MP18]

I am still getting this error i ran the command debug restart log receiver

[MP18]

I see PA is conected to Panorama and we have dedicated log collectors

[BigPalo]

are you still having this issue???