Panorama receiving logs but not showing in GUI

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Reply
Highlighted
L2 Linker

Panorama receiving logs but not showing in GUI

As the title says, Panorama receives logs just fine but will not show them in the GUI.

 

I noticed this today.  Disks shows unavailable in panorama but in the CLI they're fine.

 

Disk Status.png


Accepted Solutions
Highlighted
L1 Bithead

There have been a few issues with the elasticsearch indices causing exactly the problem you're having. (PAN-114041 is one example)

 

Make sure you're on at least PAN-OS 8.1.7 (it could have happened on a later version too,  I didn't see it skimming through though) and you will need to contact support to help you clear those indices as root access is required.

View solution in original post


All Replies
Highlighted
L3 Networker

try first restarting Mangement server, use this with a maintenance window though.

 

debug software restart process management-server

 

Highlighted
L2 Linker

I'm actually pretty convinced this is related to the elastic search process.

 

After a graceful shutdown/power up of the Panorama VM I noticed the following:

 

show system software status
...
Process  elasticsearch      restarting  (pid: -1) - Exit Code: 1
...

Every time you run this command, this process comes back with a new PID (which leads me to believe it's constantly restarting).  Each of these are within 10-15 seconds of each other:

show system software status
...
Process  elasticsearch      running  (pid: 27029)
...
show system software status
...
Process  elasticsearch      running  (pid: 27157)
...
show system software status
...
Process  elasticsearch      running  (pid: 27658)
...
show system software status
...
Process  elasticsearch      running  (pid: 27399)
...

 

Highlighted
L2 Linker

I've restarted this management server probably 10 times.  The only thing it seems to do is restart services freeing up the CPU long enough to seem stable, but then it ramps back up.  Still no logs either.

  • Restarted Panorama.
  • Restarted management server.
  • Restarted log collector.
  • Gracefully Rebooted the VM
  • Gracefully Shutdown the VM.

None of which seem to help. My hunch is the elasticsearch service is bugged out, eventually causing the CPU to spike, eventually causing the whole system to become unstable.  Nothing seems to bring logs back to the UX either.

Highlighted
L2 Linker

Gonna keep going here...I noticed that the "java" process is exceptionally high (90% - 420%).  Turns out, JAVA is used to do reporting & logging/indices.  Funny enough, those are the exact things I cannot get Panorama to do.  Panorama - JAVA Process 

 

To further my hunch about elasticsearch - it is 100% restarting constantly.  Running "show system software status | match elasticsearch" back-to-back 10 times in a row confirms it, I saw at least once where the service was "restarting (pid: -1) - Exit Code: 1".

 

Since elasticsearch is written in JAVA, I can only assume that fixing this service will fix a lot of my other issues.

Highlighted
L1 Bithead

There have been a few issues with the elasticsearch indices causing exactly the problem you're having. (PAN-114041 is one example)

 

Make sure you're on at least PAN-OS 8.1.7 (it could have happened on a later version too,  I didn't see it skimming through though) and you will need to contact support to help you clear those indices as root access is required.

View solution in original post

Highlighted
L2 Linker

We're on 9.0.9 & 9.0.9-h1.  Guess I'll see if we can skip the chit chat and get this thing reset.

Highlighted
L2 Linker

Long delay, but I finally got around to reinstalling 9.0.9 on Panorama today.  Unfortunately, it didn't fix the missing logs.  I still notice that the elasticsearch process is still restarting every second (new pid every second & caught the same exit error).  Support is still slacking, so I'll try making a point of it again.

Highlighted
L2 Linker

Finally got someone from TAC to run the "es_restart.py" script.  This does require TAC to login as root.  Logs came back almost instantly.  This issue was likely caused by a power outage, the result of which was corrupted index files for ES.  Reinstalling PANOS-9.0.9 didn't fix it, but it was certainly worth a shot!

 

Thanks!

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!