Log "Number of hints on disk has exceeded 5000 due to log forward failures."

Reply
L4 Transporter

Log "Number of hints on disk has exceeded 5000 due to log forward failures."

Hi,

 

We are receiving these logs. We would like to know what is causing these logs and how to solve it.

 

hints.JPG

 

Thanks :)

L4 Transporter

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

any idea?

L5 Sessionator

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

Hey @jesuscano

 

Check out the below thread, it seems people have resolved the issue by running the command "debug software restart process log-receiver"

 

https://live.paloaltonetworks.com/t5/General-Topics/General-PA-5220/m-p/192473#M57806

 

As for the root cause, are you running Panorama?

 

Cheers,

Luke.

L4 Transporter

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

Yes, we are running Panorama

L5 Sessionator

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

Hey @jesuscano

 

Cheers for confirming that. Did you restart the log receiver service and did it resolve the issue?

 

From what I gather, this problem is caused by the send queue being filled up when attempting to forward logs to Panorama. This can be verified by looking at the netstat output "show netstat" and looking at the "Send Queue" column for a socket open on port 10000.

 

In Panorama, there are a few best practices that we can look at:

 

1. Has a log forwarding preference list been configured? Panorama -> Collector Groups -> Device Log Forwarding

2. Is "enable redundancy across log collectors" checked?

3. Is "Forward to all collectors in the preference list" checked?

 

If options two and three are enabled, without the use of the preference list, then all logs will just be sent to one LC, and this will then be copying the logs to the other LCs anyways - causing a lot of stress. At this point the Panorama will start to throttle logs and this is when you will notice the netstat queues increasing.

 

Cheers,

Luke.

L2 Linker

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."



 

In Panorama, there are a few best practices that we can look at:

 

1. Has a log forwarding preference list been configured? Panorama -> Managed Collectors -> Device Log Forwarding



@LukeBullimore- I think that setting is under the Collector Groups, not Managed Collectors

 

Good best practices list - much appreciated!

L5 Sessionator

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

Hey @JW6224

 

Whoops yeah that was a typo, I'll correct it now. 

 

 

L4 Transporter

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

I am still getting this error i ran the command debug restart log receiver

L4 Transporter

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

I see PA is conected to Panorama and we have dedicated log collectors

 

L4 Transporter

Re: Log "Number of hints on disk has exceeded 5000 due to log forward failures."

are you still having this issue???

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!