panReadOrders crashes daily

[ChristopherMarston]

Excerpt from panReadOrders.log below.   I can manually process csv log files for machine learning but the daily process doesn't work.

Notice: Trying to get property of non-object in /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicTask.php on line 14
Checking: PeriodicLogCollectorCompacter
Thu, 11 Nov 2021 15:41:50 +0000 Start Task
Checking CSV logs from device(s) 000000000000000 ,111111111111111 ,2222222222222222 ,333333333333333

Notice: Trying to get property of non-object in /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php on line 104

Notice: Trying to get property of non-object in /var/www/html/libs/common/device/CsvFilesProcesser.php on line 130

Fatal error: Uncaught TypeError: Argument 3 passed to accessControl() must be of the type integer, null given, called in /var/www/html/libs/common/device/CsvFilesProcesser.php on line 132 and defined in /var/www/html/userManager/API/accessControl_CLI.php:23
Stack trace:
#0 /var/www/html/libs/common/device/CsvFilesProcesser.php(132): accessControl('any', Array, NULL)
#1 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php(104): PaloAltoNetworks\Sentinel\Devices\CsvFilesProcesser->processCSVLogs(NULL)
#2 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php(80): PeriodicLogCollectorCompacter->processCSVLogs(Array)
#3 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/PeriodicLogCollectorCompacter.php(31): PeriodicLogCollectorCompacter->execute()
#4 /var/www/html/console/messaging/classes/PaloAltoNetworks/Processes/ClassLoader.php(69): PeriodicLogCollectorCompacter->start()
#5 /var/www/html/console/messaging/classes in /var/www/html/userManager/API/accessControl_CLI.php on line 23

[azuniga]

Can you send over a screenshot of your folder permissions for the PALogs and the folder within PALogs.

[ChristopherMarston]

[azuniga]

Can you also show the PALogs folder permissions as well?

[ChristopherMarston]

That is represented by the first line "." ( current directory ).  The second line ".." ( parent directory, in this case the root directory )

[azuniga]

The PALogs folder itself should also be owned by www-data:expedition otherwise it cannot write to the folder within PALogs.

[ChristopherMarston]

I change the group for the PALogs folder and did a recursive rights to 775.  Thanks for the info.  Should know tomorrow if that fixed the issue.

[azuniga]

Thank you for the update and good luck with your migration.

[ChristopherMarston]

Unfortunately the panReadOrders crashed again after the group ownership/permission changes to PALogs folder.

[azuniga]

Hello @ChristopherMarston 

Something could have become corrupted, can you run the following commands and show the output?

history | grep upgrade

[ChristopherMarston]

I don't believe any OS or package upgrades have been done since the original install of Ubuntu 16.04.7 LTS

ubuntu@host:/home/userSpace$ history | grep upgrade
1101 history | grep upgrade

[azuniga]

Hello @ChristopherMarston 

Typically the cause is rabbitmq so if you perform the following steps it can resolve it.

apt-get remove rabbitmq-server && apt-get purge rabbitmq-server

then reinstall it

     apt-get install rabbitmq-server

Sudo rabbitmq-server start

[ChristopherMarston]

Performed the steps you recommended and still panReadOrders crashes when running on schedule.  It may not be related but I had always added firewalls via Panorama device and then enabled ML with-in each device.  I recently tried to enable ML on the Panorama device and then add it to a project selecting specific Device Group in the collector setup.  The analyze stage would always hang so I gave up but it seems this problem has been occurring since then.  I've since deleted the project I was using to test and went back to Device / Panorama and removed the path, auto-process, logs from syslog and saved the settings.  It may be coincidental but it seems these issues started when tried doing ML on a Device Group as described above.

[ChristopherMarston]

panReadOrders is still crashing requiring that I process logs manually/daily.  Any help is appreciated.

[ChristopherMarston · Accepted solution]

I deleted the parquet files "SETTINGS - M. LEARNING - DATA ANALYSIS STRUCTURES FOLDER - DELETE Connections.parquet FILES" and then manually imported the last 30 Days of logs and the scheduled log import has been working for about a week now.