Questions) Missing Panorama Log

[future]

When creating the Security Policy Rule, 'Log at Session Start/End' was all selected as Actions.

After this, when I check the log in Panorama, only the End Log is visible and the Start Log is not visible.
Also, sometimes this logs are not visible.

In Panorama, there are cases where the log cannot be seen or some logs (Session Start) are not visible like this.
What causes this to happen?

Could there be a lot of traffic logs and some logs might be missing from Panorama?
Has anyone had a similar experience?

[PavelK]

Thank you for the post @future

troubleshooting Panorama missing logs is complex and requires more input from your side.

First thing, could you confirm what PAN-OS version you are running on Firewall? If you are in 9.1 release, then I would recommend upgrade to 9.1.14. In this version there is a bug fix: PAN-185616 

https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-release-notes/pan-os-9-1-addressed-issues/pan-os...

I confirmed with TAC that this is not only limited to syslog, but also affects sending logs to Panorama.

Could you also check Panorama log collector side to confirm from CLI whether there are any Fails:

debug log-collector log-collection-stats show incoming-logs | match Fails

If the number of Fails is anything other than 0, this indicates that some logs are failing to be written to disks.

Could you also confirm what PAN-OS version you are running on Panorama and whether you are using dedicated log collectors or local log collector.

Kind Regards

Pavel

[future]

Thanks for the very kind reply.

I will check further base your advice.
I was impressed by the positive and complete response.
Thank you very much.

After additional confirmation, we will deliver happy news when it is complete.😀

[PavelK]

Thank you for reply and your comment @future

I am sure you must have a good reason to keep both "Log at Session Start" and "Log at Session End" enabled. Just in case here is a KB: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Clt5CAC where Palo Alto does not recommend to keep both options enabled unless you are performing a troubleshooting.

After you perform the upgrade to 9.1.14 and you are still experiencing an issue where logs are available on Firewall locally, but missing in Panorama, then if you see any Fails in the output: "debug log-collector log-collection-stats show incoming-logs | match Fails" there are at least 2 possible root causes I can think of:

- If you have distributed environment with multiple log collectors and there is a latency between log collectors more than 10ms, this might result log loss. Here is corresponding KB: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CmUnCAK

- High logging rate from Firewall side that will cause failing to write logs to disk. This will however require more investigation.

Kind Regards

Pavel