topic Re: panReadOrders crashes daily in Expedition Discussions

panReadOrders crashes daily

ChristopherMarston — Thu, 11 Nov 2021 16:00:01 GMT

Excerpt from panReadOrders.log below. I can manually process csv log files for machine learning but the daily process doesn't work.

Notice: Trying to get property of non-object in /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicTask.php on line 14
Checking: PeriodicLogCollectorCompacter
Thu, 11 Nov 2021 15:41:50 +0000 Start Task
Checking CSV logs from device(s) 000000000000000 ,111111111111111 ,2222222222222222 ,333333333333333

Notice: Trying to get property of non-object in /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php on line 104

Notice: Trying to get property of non-object in /var/www/html/libs/common/device/CsvFilesProcesser.php on line 130

Fatal error: Uncaught TypeError: Argument 3 passed to accessControl() must be of the type integer, null given, called in /var/www/html/libs/common/device/CsvFilesProcesser.php on line 132 and defined in /var/www/html/userManager/API/accessControl_CLI.php:23
Stack trace:
#0 /var/www/html/libs/common/device/CsvFilesProcesser.php(132): accessControl('any', Array, NULL)
#1 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php(104): PaloAltoNetworks\Sentinel\Devices\CsvFilesProcesser->processCSVLogs(NULL)
#2 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php(80): PeriodicLogCollectorCompacter->processCSVLogs(Array)
#3 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php(31): PeriodicLogCollectorCompacter->execute()
#4 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/ClassLoader.php(69): PeriodicLogCollectorCompacter->start()
#5 /var/www/html/console/messaging/classes in /var/www/html/userManager/API/accessControl_CLI.php on line 23

Re: panReadOrders crashes daily

azuniga — Thu, 11 Nov 2021 16:28:27 GMT

Can you send over a screenshot of your folder permissions for the PALogs and the folder within PALogs.

Re: panReadOrders crashes daily

ChristopherMarston — Thu, 11 Nov 2021 17:34:43 GMT

Re: panReadOrders crashes daily

azuniga — Thu, 11 Nov 2021 17:51:57 GMT

Can you also show the PALogs folder permissions as well?

Re: panReadOrders crashes daily

ChristopherMarston — Thu, 11 Nov 2021 19:11:44 GMT

That is represented by the first line "." ( current directory ). The second line ".." ( parent directory, in this case the root directory )

Re: panReadOrders crashes daily

azuniga — Thu, 11 Nov 2021 19:27:24 GMT

The PALogs folder itself should also be owned by www-data:expedition otherwise it cannot write to the folder within PALogs.

Re: panReadOrders crashes daily

ChristopherMarston — Thu, 11 Nov 2021 20:58:38 GMT

I change the group for the PALogs folder and did a recursive rights to 775. Thanks for the info. Should know tomorrow if that fixed the issue.

Re: panReadOrders crashes daily

azuniga — Thu, 11 Nov 2021 21:31:49 GMT

Thank you for the update and good luck with your migration.

Re: panReadOrders crashes daily

ChristopherMarston — Fri, 12 Nov 2021 16:52:41 GMT

Unfortunately the panReadOrders crashed again after the group ownership/permission changes to PALogs folder.

Re: panReadOrders crashes daily

azuniga — Fri, 12 Nov 2021 19:06:36 GMT

Hello @ChristopherMarston

Something could have become corrupted, can you run the following commands and show the output?

history | grep upgrade

Re: panReadOrders crashes daily

ChristopherMarston — Fri, 12 Nov 2021 20:26:54 GMT

I don't believe any OS or package upgrades have been done since the original install of Ubuntu 16.04.7 LTS

ubuntu@host:/home/userSpace$ history | grep upgrade
1101 history | grep upgrade

Re: panReadOrders crashes daily

azuniga — Fri, 12 Nov 2021 23:13:54 GMT

Hello @ChristopherMarston

Typically the cause is rabbitmq so if you perform the following steps it can resolve it.

apt-get remove rabbitmq-server && apt-get purge rabbitmq-server

then reinstall it

apt-get install rabbitmq-server

Sudo rabbitmq-server start

Re: panReadOrders crashes daily

ChristopherMarston — Sun, 14 Nov 2021 03:33:30 GMT

Performed the steps you recommended and still panReadOrders crashes when running on schedule. It may not be related but I had always added firewalls via Panorama device and then enabled ML with-in each device. I recently tried to enable ML on the Panorama device and then add it to a project selecting specific Device Group in the collector setup. The analyze stage would always hang so I gave up but it seems this problem has been occurring since then. I've since deleted the project I was using to test and went back to Device / Panorama and removed the path, auto-process, logs from syslog and saved the settings. It may be coincidental but it seems these issues started when tried doing ML on a Device Group as described above.

Re: panReadOrders crashes daily

ChristopherMarston — Fri, 19 Nov 2021 15:52:44 GMT

panReadOrders is still crashing requiring that I process logs manually/daily. Any help is appreciated.

Re: panReadOrders crashes daily

ChristopherMarston — Tue, 30 Nov 2021 18:37:56 GMT

I deleted the parquet files "SETTINGS - M. LEARNING - DATA ANALYSIS STRUCTURES FOLDER - DELETE Connections.parquet FILES" and then manually imported the last 30 Days of logs and the scheduled log import has been working for about a week now.