FWs no longer forwarding logs to Panorama

[drewdown]

Not sure what happened but it seems that all my firewalls stop sending logs to panorama (local log collector).   I have a ticket open with PAN support but not really getting anywhere.  

Recently upgraded PANORAMA to 11.1.13-h3 and added additional collector disks to PANORAMA.   Also forwarded logs from PANORAMA log collector to our syslog server and at some point it seems all FWs just stopped sending any logs to PANORAMA even though they have a log profile set to do so on a large number of policies.   This worked fine up until recently but I can't say for sure when it broke but its, without a doubt, broken as all get up.    I don't see anything over 514 from the FWs to panorama either.  So they aren't sending anything I just have no idea why.  

Has anyone seen this before?  from one of the local FWs:

From PANORAMA:

All firewalls being managed by panorama show the same N/A as below.  They are all connected to panorama so its not a disconnected fw from panorama issue.

What is strange from panorama and screenshot above is the source IP is showing the FW's serial number vs its actual IP address.   Does anyone know what could be causing this?  Its some weird config issue hit a bug or what I don't even know.  

This is the log forwarding profile which has been in use for a long time:

[A.DeGracia059588]

Log into one of the FWs and pcap the output. Make sure its sending the data.

Then log into the Fw protecting Pano and pcap the input and make sure its at least reaching that part of the network.

Potentially pano service has stalled and service port is not listening. 

[TomYoung]

Hi @drewdown ,

The same command on my Panorama also shows the S/N for the Source IP.  I think that is normal.  I don't remember if I ran into the same issue, but I have seen the logs stop forwarding on rare occasions.  I had to be very methodical and go through a lot of steps to fix.  Here is a good place to start.  https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000HE48CAG

You may also need to review the steps in the Configure Log Forwarding doc as well.

Thanks,

Tom