PanOrders Agent is stopped. Cant restart

[NateMiles]

My expedition server ran out of space because I was using the ML and sending logs to the expedition server.

Cleared that space and ever since then the PanOrders agent will not start.

this is the output from the panReadOrders.log file

------------------------------------------------------------
Wed Jul 18 15:51:51 CDT 2018 : Starting readOrders.php
------------------------------------------------------------

Fatal error: Uncaught ErrorException: stream_socket_client(): unable to connect to tcp://localhost:5672 (Connection refused) in /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php:141
Stack trace:
#0 [internal function]: PhpAmqpLib\Wire\IO\StreamIO->error_handler(2, 'stream_socket_c...', '/var/www/html/l...', 141, Array)
#1 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php(141): stream_socket_client('tcp://localhost...', 111, 'Connection refu...', 3, 4, Resource id #113)
#2 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(205): PhpAmqpLib\Wire\IO\StreamIO->connect()
#3 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(189): PhpAmqpLib\Connection\AbstractConnection->connect()
#4 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AMQPStreamConnection.php(60): PhpAmqpLib\Connection\AbstractConnection->__construct('guest', 'guest', '/', false, 'AMQPLAIN', in /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/IO/StreamIO.php on line 141

I manualy edited the ports.conf file in apache2 to Listen 5672 and then got this

Wed Jul 18 16:11:21 CDT 2018 : Starting readOrders.php
------------------------------------------------------------

Fatal error: Uncaught PhpAmqpLib\Exception\AMQPTimeoutException: The connection timed out after 3 sec while awaiting incoming data in /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/AMQPReader.php:132
Stack trace:
#0 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/AMQPReader.php(148): PhpAmqpLib\Wire\AMQPReader->wait()
#1 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/AMQPReader.php(106): PhpAmqpLib\Wire\AMQPReader->rawread(7)
#2 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(522): PhpAmqpLib\Wire\AMQPReader->read(7)
#3 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Connection/AbstractConnection.php(570): PhpAmqpLib\Connection\AbstractConnection->wait_frame(3)
#4 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php(225): PhpAmqpLib\Connection\AbstractConnection->wait_channel(0, 3)
#5 /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Channel/AbstractChannel.php( in /var/www/html/libs/vendor/php-amqplib/php-amqplib/PhpAmqpLib/Wire/AMQPReader.php on line 132

Any help would be great

[NateMiles · Accepted solution]

Thank you for pointing that out.

Following your post I simply removed the rabbitmq-server

as root

     apt-get remove rabbitmq-server && apt-get purge rabbitmq-server

then reinstalled it

     apt-get install rabbitmq-server

Verified fixed now.

Thanks again

[dgildelaig]

That port is not used by Apache, but by rabbitmq.

If you make Apache bind to that port, the messaging system that the agent uses won't work.

For some reason, I would say that the issue was that rabbitmq stopped working on your system, therefore the connection was refused.

This may be that, through the cleaning, the configuration for rabbitmq got corrupted, some files for the service are missing, or simply it also got hung due to lack of space (and a reboot after the cleaning would have been enough)

[NateMiles · Accepted solution]

Thank you for pointing that out.

Following your post I simply removed the rabbitmq-server

as root

     apt-get remove rabbitmq-server && apt-get purge rabbitmq-server

then reinstalled it

     apt-get install rabbitmq-server

Verified fixed now.

Thanks again

[stevenjwilliams83]

Cant login as root so how did you accomplish this?

[sjanita]

after logging in as the 'expedition' user issue:

$sudo su

enter the password for the expedition user.

[Blake_Wofford]

Unfortunately that did not fix it.  Here's what the uninstall and install had to say:  Failed to start RabbitMQ broker.

Last login: Mon Aug 24 16:22:48 2020 from 198.18.150.146
expedition@Expedition:~$ sudo apt-get remove rabbitmq-server && apt-get purge rabbitmq-server
[sudo] password for expedition:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
erlang-asn1 erlang-base erlang-corba erlang-crypto erlang-diameter erlang-edoc erlang-eldap erlang-erl-docgen erlang-eunit erlang-ic erlang-inets erlang-mnesia erlang-nox erlang-odbc erlang-os-mon erlang-parsetools erlang-percept
erlang-public-key erlang-runtime-tools erlang-snmp erlang-ssh erlang-ssl erlang-syntax-tools erlang-tools erlang-webtool erlang-xmerl libodbc1 libsctp1
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
rabbitmq-server
0 upgraded, 0 newly installed, 1 to remove and 28 not upgraded.
After this operation, 5,830 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 114913 files and directories currently installed.)
Removing rabbitmq-server (3.6.15-1) ...
Processing triggers for man-db (2.7.5-1) ...
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
expedition@Expedition:~$ sudo apt-get install rabbitmq-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
rabbitmq-server
0 upgraded, 1 newly installed, 0 to remove and 28 not upgraded.
Need to get 5,025 kB of archives.
After this operation, 5,830 kB of additional disk space will be used.
Get:1 https://www.rabbitmq.com/debian testing/main amd64 rabbitmq-server all 3.6.15-1 [5,025 kB]
Fetched 5,025 kB in 4s (1,112 kB/s)
Selecting previously unselected package rabbitmq-server.
(Reading database ... 114706 files and directories currently installed.)
Preparing to unpack .../rabbitmq-server_3.6.15-1_all.deb ...
Unpacking rabbitmq-server (3.6.15-1) ...
Processing triggers for systemd (229-4ubuntu21.28) ...
Processing triggers for ureadahead (0.100.0-19.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up rabbitmq-server (3.6.15-1) ...
Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
● rabbitmq-server.service - RabbitMQ broker
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2020-08-24 16:24:51 CDT; 4ms ago
Process: 48614 ExecStop=/bin/sh -c while ps -p $MAINPID >/dev/null 2>&1; do sleep 1; done (code=exited, status=0/SUCCESS)
Process: 48500 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 48301 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILURE)
Main PID: 48301 (code=exited, status=1/FAILURE)
Status: "Exited."

Aug 24 16:24:51 Expedition systemd[1]: Failed to start RabbitMQ broker.
Aug 24 16:24:51 Expedition systemd[1]: rabbitmq-server.service: Unit entered failed state.
Aug 24 16:24:51 Expedition systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
dpkg: error processing package rabbitmq-server (--configure):
subprocess installed post-installation script returned error exit status 1
E: Sub-process /usr/bin/dpkg returned an error code (1)
expedition@Expedition:~$

[lychiang]

@Blake_Wofford 

Could you try run commands listed in the below thread:

https://live.paloaltonetworks.com/t5/expedition-discussions/rabbitmq-not-properly-starting-failing/t...

[Blake_Wofford]

I ran the commands in the last article you referred me to.  Unfortunately, it still hasn't fixed the problem.  We are going to re-install with a new OVA file to resolve this issue.  Thanks for all your help!

[azuniga]

@Blake_Wofford 

If you continue to have issues let us know and we can start a zoom session.