topic Re: "Number of hints on disk has exceeded 5000 due to log forward failures." in Log Forwarding Discussions

"Number of hints on disk has exceeded 5000 due to log forward failures."

RicardoNeira — Mon, 29 Oct 2018 12:04:44 GMT

Hi,

I have a client (PA 5220 version 8.0.9) who continuously (every hour) is getting this error message in Monitor -> System: "Number of hints on disk has exceeded 5000 due to log forward failures."

At first we thought it was due to the parameter configured under Device -> Setup -> Management -> Logging and Reporting Settings -> Max Rows in User Activity Report since the value was 5000, but we are no longer sure

We also suspect that it was due to the maximum number of user authentication errors and / or external log elements and we limited the storage of this type of logs. But the alarm is still active

We do not know what is the origin of this error to be able to solve it and let it appear.

Some idea of what is causing this error message and how to fix it.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

Navigator — Mon, 10 Dec 2018 07:51:07 GMT

Hi ricardo,
I know this may seem old but would you please provide us with the resolution for your issue?

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

RicardoNeira — Fri, 14 Dec 2018 11:59:17 GMT

I tried restarting the log receiver from the root but this didn´t solve the problem.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

MannyG — Wed, 16 Jan 2019 05:23:32 GMT

Ricardo,

I know it's been over a month now, but were you able to resolve this issue?

I had a 5220 hardware failure on my active/standby pair. I replaced the failed firewall and synced/copied the standby config to the active (my active is the one that died). Now I'm getting these alerts.

My device is registered, license transferred, OS version are same on both firewalls, license number was replaced in Panorama from old to new. Not sure what the deal is.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

RicardoNeira — Wed, 16 Jan 2019 07:42:44 GMT

Hi,

Not yet. I see that it is possible this problem is relacionated with Panorama. In my customer Panorama was deactivated but not in the config of the Palo Alto so I was expecting they delete this.

After that and if it don´t work. I´ll apply again the "debug software restart process log-receiver" command.

Regards

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

nagarjuna.b — Thu, 14 Feb 2019 03:37:40 GMT

Not sure if you have already figured this out, if not here is my suggestion and what I did to fix this thing few days back.

I did verify this on my firewall and I see logs are not forwarding to Panoramaa

devicename>debug log-receiver rawlog_fwd statistics global show

There were many drops in the output of the command.

made sure logs log settings are configured to forward the logs to Panorama

but, on the Panorama, under log collector groups we haven't add the firewall under device log forwarding list. that fixed the issue. in fact not immediately because the hints count is something that clear off only when all the logs that were stored on the hints were forwarded to panorama. it will send one log per sencond. the maximum hint count is 20000 by default, but device generate high priority system log when it exceeds 5000. I just waited until until all logs on the hints were written to panorama, however if you want you can clear off the hint count with

devincename> debug log-receiver rawlog_fwd clear hints-all

Hope this helps.

Best regards,

Nagarjuna

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

kenvizena — Fri, 22 Feb 2019 22:09:53 GMT

This has popped up two or three times for me, in the first two it was running a fw that was a higher version than Panorama. My most recent example was running an older version of 8.0.x log collectors against a 8.1.x Panorama and 8.1.x FW.

I would do a show logging-status to see if there is a misconfiguration and make note of the addresses.

Take the results from the prior command:

show netstat all yes | match 10.x.x.x

It should look something like this:

tcp your.firewall.com:50000 10.x.x.x:pan-panorama establshed

If that looks fine, then I would logon to the Panorama CLI and run this command:

show netstat all yes | match 3978 (may be 3798, not at a console)

If it shows an active connection and you are running the exact same version on the fw, panorama or log collectors I would open a case with PA.

I would verify the the time on all devices match and if using log collectors to make sure the dynamic updates are working and all are the same version, otherwise collation will not allow the logs to be processed.

You can try and run this from Panorama to see if it can restart the connection.

request log-fwd-ctrl device SERIALNUMBER start-from-lastack
request log-fwd-ctrl device SERIALNUMBER action stop
request log-fwd-ctrl device SERIALNUMBER action live
request log-fwd-ctrl device SERIALNUMBER action start

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClFCCA0
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClXACA0

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

RicardoNeira — Tue, 09 Apr 2019 06:31:12 GMT

Finally the problem was solved by TAC

As we don´t have an Panorama anymore they activate the HIP Mach parameter: "hipmatch-any" to Panorama (under Device--> Log settings).

After that we check with "debug management-server rawlog_fwd show hint-state" that we have a lot of records in

Number of hints on disk (over 19200) so they clear all this hints with "debug management-server rawlog_fwd clear hints-all" untill reach 0.

We deativate the HIP Match to the Panorama and commit

After that we don´t have more alarms

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

jim.smith — Thu, 21 Oct 2021 17:11:28 GMT

While this is an old thread... wanted to share current experience. PA220's - No Panorama or remote logging enabled. Upgraded to 10.1.2 and within a couple days received the Hints on Disk error. Referencing KB > https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PPWtCAO found all "Debug management-server rawlog_fwd" commands result in "Invalid Syntax".

Additionally, the "debug log-receiver rawlog_fwd clear hints-all" ONLY removes 64 entries at a time. Rinse and repeat to achieve '0'.

Any ideas on root cause for Hints on disk and why they do not auto-purge? Since zero-ing out, the number has been growing.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

jim.smith — Thu, 21 Oct 2021 17:28:35 GMT

Typical... no sooner than posting I find additional info.

Found the following CLI info for v10.1 >> https://docs.paloaltonetworks.com/pan-os/10-1/pan-os-cli-quick-start/cli-command-hierarchy-for-pan-os-101/pan-os-101-cli-ops-command-hierarchy

In here there is "debug log-receiver rawlog_fwd set hints-expiration-duration <0-846000>" (default is set to '0')

Asking support for "best-practice" as this is a large range and I'm unclear on the affect of hints in the current environment.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

RobertShawver — Fri, 03 Dec 2021 18:36:11 GMT

Sorry to add to the way back machine, but in case someone comes across this like I did, the fix for me was to uncheck the "Enable log redundancy across collectors".

Panorama > Collector Groups > [your collector group name] > General > uncheck the "Enable log redundancy across collectors".

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

DanielDarmawi — Wed, 06 Sep 2023 00:25:24 GMT

Hi Ricardo,

Is there any impact on clearing the hints count?

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Tue, 28 May 2024 16:11:18 GMT

Also, if you can restart the log receiver process on Firewall and then delete the log-collector preference list it works fine.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

NickoKristian — Wed, 16 Oct 2024 07:58:07 GMT

Hi Sir,

Can you share us on how to restart this?

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Wed, 16 Oct 2024 19:04:48 GMT

Use the commands below

debug software restart process log-receiver

delete log-collector preference-list

This should fix the issue.

Regards

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

NetworkingKimitec — Thu, 17 Oct 2024 08:54:35 GMT

Hi Jim, I'm in the same scenario.

Did you finally solve the problem and did you find the affectation of the hints?

thanks you in advance.

BR.

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Tue, 22 Oct 2024 13:55:05 GMT

@NetworkingKimitec

Please use below commands to fix the issue

debug software restart process log-receiver

delete log-collector preference-list

Regards

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

NickoKristian — Fri, 25 Oct 2024 01:10:38 GMT

Hi @MP18 ,

Would running these commands incur downtime on the services?

Regards

Nicko

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

MP18 — Fri, 25 Oct 2024 01:50:30 GMT

@NickoKristianNickoKristian These commands are safe to run anytime.

Regards

Re: "Number of hints on disk has exceeded 5000 due to log forward failures."

NetworkingKimitec — Wed, 30 Oct 2024 10:16:53 GMT

I have executed the comands. but the problem persists:

any ideas?

Thank you in advance.

BR.