Multiple Issues, ran out of space

Reply
Highlighted
L1 Bithead

Multiple Issues, ran out of space

We filled up the drive by doing scheduled log exports, we then added a new disk and moved log exports to it, the only thing I had to fix was PanReadOrders was showing as stopped and when we tried to start it, it would just stop again. So I found an article where it was recommended to remove and re-add Rabbitmq and that got the Jobs and Task manager to start again

 

https://live.paloaltonetworks.com/t5/expedition-discussions/panorders-agent-is-stopped-cant-restart/...

 

But now things don't seem well, I tried to pull down Contents again in Devices and everything shows as Updating or Downloading, and there are 4 pending jobs in Task Manager.

 

I checked the panReadOrders.log file around the time when it ran out of space and ever since then the bottom errors about a non-object are just repeating for 99% of the file...

 

Also in Projects things don't seem to be working, ML and Rule Enhancement both seem to think no devices are added, and under Devices the Traffic tab never shows anything.

 

Is Expedition hosed?

Errors.PNG

 

Highlighted
L1 Bithead

When this issue happened to us, we tried to replace the RabbitMQ as well, but that didn't solve our problem.  Once we saw that this was going to be a much  more involved process for restoring our expedition server, I went back to the VMWare snapshot that we had and restored from there.  That was much more time-efficient for us. 

 

However, even if we didn't have the snapshot, we would have just re-installed via the OVA fiile.  Most of what we currently had in expedition was research related and could be re-created fairly easily.  I know that doesn't really solve the issue, but it's the path that we took.

Highlighted
L4 Transporter

Hello @BRS-Alex 

 

You can always check your volume with "df -k" to confirm where most of your space is being consumed and confirm the size is no longer adequate. Also once you confirm that my recommendation would be to re-size the VM, if you are unable to do that then I would mount another drive to the VM.

Highlighted
L1 Bithead

We did just that, and mounted another drive and resolved the space issue, then came the Rabbitmq issue and since then the Task Scheduler seems to be broken.

Tags (1)
Highlighted
L1 Bithead

I was thinking that would be my next course of action, but before going that route I thought I'd throw it out there, maybe there was another step in restoring RabitMQ that I'm not aware of... but yeah it's a good bit of work we would be throwing away since the last snapshot.

Highlighted
L4 Transporter

@BRS-Alex 

 

You can try the following commands to fix rabbitmq

 

apt-get remove rabbitmq-server && apt-get purge rabbitmq-server
 
then reinstall it
     apt-get install rabbitmq-server

Sudo rabbitmq-server start

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!