topic Re: Multiple Issues, ran out of space in Expedition Discussions

Multiple Issues, ran out of space

BRS-Alex — Thu, 10 Sep 2020 19:15:21 GMT

We filled up the drive by doing scheduled log exports, we then added a new disk and moved log exports to it, the only thing I had to fix was PanReadOrders was showing as stopped and when we tried to start it, it would just stop again. So I found an article where it was recommended to remove and re-add Rabbitmq and that got the Jobs and Task manager to start again

https://live.paloaltonetworks.com/t5/expedition-discussions/panorders-agent-is-stopped-cant-restart/td-p/222870

But now things don't seem well, I tried to pull down Contents again in Devices and everything shows as Updating or Downloading, and there are 4 pending jobs in Task Manager.

I checked the panReadOrders.log file around the time when it ran out of space and ever since then the bottom errors about a non-object are just repeating for 99% of the file...

Also in Projects things don't seem to be working, ML and Rule Enhancement both seem to think no devices are added, and under Devices the Traffic tab never shows anything.

Is Expedition hosed?

Re: Multiple Issues, ran out of space

Blake_Wofford — Thu, 10 Sep 2020 19:20:30 GMT

When this issue happened to us, we tried to replace the RabbitMQ as well, but that didn't solve our problem. Once we saw that this was going to be a much more involved process for restoring our expedition server, I went back to the VMWare snapshot that we had and restored from there. That was much more time-efficient for us.

However, even if we didn't have the snapshot, we would have just re-installed via the OVA fiile. Most of what we currently had in expedition was research related and could be re-created fairly easily. I know that doesn't really solve the issue, but it's the path that we took.

Re: Multiple Issues, ran out of space

azuniga — Thu, 10 Sep 2020 20:25:22 GMT

Hello @BRS-Alex

You can always check your volume with "df -k" to confirm where most of your space is being consumed and confirm the size is no longer adequate. Also once you confirm that my recommendation would be to re-size the VM, if you are unable to do that then I would mount another drive to the VM.

Re: Multiple Issues, ran out of space

BRS-Alex — Thu, 10 Sep 2020 20:44:17 GMT

We did just that, and mounted another drive and resolved the space issue, then came the Rabbitmq issue and since then the Task Scheduler seems to be broken.

Re: Multiple Issues, ran out of space

BRS-Alex — Thu, 10 Sep 2020 20:45:37 GMT

I was thinking that would be my next course of action, but before going that route I thought I'd throw it out there, maybe there was another step in restoring RabitMQ that I'm not aware of... but yeah it's a good bit of work we would be throwing away since the last snapshot.

Re: Multiple Issues, ran out of space

azuniga — Thu, 10 Sep 2020 21:48:35 GMT

@BRS-Alex

You can try the following commands to fix rabbitmq

apt-get remove rabbitmq-server && apt-get purge rabbitmq-server

then reinstall it
apt-get install rabbitmq-server

Sudo rabbitmq-server start