Panorama doesnt show traffic or threat logs

[Udupi]

Hello Everyone,

I am in the middle of trying to fix an issue with Panorama unable to view traffic or threat logs. Here's the environment,

7 or 8 firewalls forwarding logs to a dedicated collector/group which is just 1 m-100 appliance running in logger mode.

We recently encountered this problem after which, restarted the log and management daemons on the collector and starting seeing the logs on collector. We also have log collector setup to forwarded to an external server which was receiving the latest logs too.

However Panorama still doesnt show any recent logs and last logs seen are 2 weeks ago. Below outputs show the collector is sending config and system 'only' which are indeed visible.

Source Daemon : unknown
Connection Id : 00XXXXXXXXX69
Log rate: 0
Log rate: 0
config 2020/06/15 08:51:41 18 2020/06/15 08:51:40 0
system 2020/10/12 07:32:16 627390 2020/10/12 07:32:00
threat 0
traffic 0
hipmatch 0
gtp 0
userid 0
iptag 0
auth 0
sctp 0

Collector is 'in-sync'  and connected. Attempts to restart the management daemon on panorama did not help, still trying to wonder why recent traffic or threat logs dont show up.

Edit: - When i go to collector configuration on Panorama GUI and hit statistics, no data is shown.

[kiwi]

Hi @Udupi ,

From your description I'm thinking the following needs to be checked:
1- FW sending logs ?  Can you check from the FWs perspective (show logging status) if the ACKs are coming to FW ?
2- LC not ingesting logs (process down/connection flapping/Throttling etc)
3- Panorama query to LC not working (check reportd)

Can you narrow down which side is the issue first ? With the current data, anything seems possible.

Hope this helps.

-Kiwi.

[Udupi]

Hi there,

Thanks for the inputs, totally forgot to reply back.

1. Yes the firewall is sending logs to collector.

2. Collector receiving the logs is also forwarding it successfully to external syslog/SIEM server which rules out firewall(s) here.

3. Panorama query is the problem i am currently troubleshooting. While its able to query log from LC prior to 2 weeks, any latest logs aren't seen. What sorta information should i be looking for under reportd? any specific hints, issues i should be concerned about?

[MP18]

@Udupi 

Are you able to see logs in CLI of Panorama?

On Panorama GUI under traffic logs click on refresh logs then run the below command from CLI

Try this command tail follow yes mp-log reportd.log

Look for below info 

* connect to 127.0.0.1 port 9200 failed: Connection refused
* Failed to connect to 127.0.0.1 port 9200: Connection refused
* Closing connection 0

Do you see above  in logs?

IF yes then you may need to restart Panorama .

Regards

[rsaloskar]

I got the same logs.. but instead of restarting the Panorama, restarting the management-server process resolved the issue.

Used 'debug software restart process management-server'

[Muqtar_Khan · Accepted solution]

Last week I upgraded new latest version to 10.2.2h2. I lost ACC and Monitor logs. It's blank. Palo Alto TAC and I tried every cmds but still no results.  Palo Alto TAC says they're researching not sure how long it takes... 😞

[caraym]

Hi @Muqtar_Khan, did you get this latest issue resolved? I am having the exact same problem. Would you be able to share the solution if any?

[Muqtar_Khan]

Hello @caraym , Yes. My Panorama issue got solved. TAC is still doing an investigation and I can't wait. So, I took a decision to downgrade to a lower preferred version in which I look into OS Article and downgrade to 10.1.6-h6 from 10.2.2-h2.

I don't know what version you upgraded to? 

Support PAN-OS Software Release Guidance | Palo Alto Networks

[mark_petitta]

Restarting the management server process on Panorama didn't seem to resolve the issue for us but then we rebooted the dedicated log collector and that fixed the problem and logs began to appear in Panorama. Our devices are running 10.1.11.