Expedition csv logs stuck in pending

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Expedition csv logs stuck in pending

L1 Bithead

Hi everyone,

I have added firewall logs from our Palo Alto 5000 series to the Expedition VM /PALogs . I have copied the orginal .csv as a duplicate with root as the owner and the original with expedition as the owner. Both files appear in Devices > M.LEARNING. When I run Process Files the job remains in pending and nothing happens. Any ideas what the issue may be?

Screen Shot 2018-06-26 at 10.35.44 AM.png
Screen Shot 2018-06-26 at 10.37.27 AM.pngScreen Shot 2018-06-26 at 10.36.50 AM.png

51 REPLIES 51

L2 Linker

Most likely the webserver does not have permission - specifically for one owned by root.

 

Try this from root dir

 

sudo chown -R www-data:www-data PALogs/

Ok so I tried running:

sudo chown -R www-data:www-data PALogs/


Screen Shot 2018-06-26 at 12.18.09 PM.png

Screen Shot 2018-06-26 at 12.20.55 PM.png

Unfortunatley the job is still in pending and not procesing. any other thoughts?

Thanks for the assistance.

Where is your machine learning path set up and does www-data have proper permissions in that folder to output the files after they have been processed?

This is what I have configured for the machine learning paths:

Screen Shot 2018-06-26 at 12.58.37 PM.pngScreen Shot 2018-06-26 at 12.58.09 PM.pngScreen Shot 2018-06-26 at 12.57.52 PM.png

That looks good.

 

Just thinking outloud and giving a few possible solutions:

 

Run these:

 

sudo apt -o "Acquire::AllowInsecureRepositories=true" -o "Acquire::AllowDowngradeToInsecureRepositories=true" update
sudo apt upgrade
sudo apt upgrade
-------------------------
Then edit /etc/php/7.0/apache2/php.ini
line mysqli.reconnect = off (change to on)
Then edit /etc/php/7.0/cli/php.ini
line mysqli.reconnect = off (change to on)
-------------------------------
sudo service apache2 restart
sudo php /var/www/html/bin/updates/updateSQL.php

Thanks again for all of your help. I tried the suggestions you made and was still unable to process the csv. It remains on pending, I've also tried to remove the process and start over again but yield the same results.

Thanks

Then try to export that csv and see if it is valid log export, maybe it got corrupted in the process and expedition does tell you that. 

Ok so I re-exported the csv then scp'd it over to expedition, still no luck. I then deleted the vm and re-downloaded and re-setup a new vm, same issue. Maybe there is a bug in the newest release?

After executing this 

sudo chown -R www-data:www-data PALogs/ 

My firewalls no longer have permission to SCP log files to Expedition.  

 

What should the overall  permissions be for the logs folder?

Esfeld,

 

Permissions depend on which directory you use for machine learning and which one for logs. In case they are the same, you would have to change the permissions every time you try to process or SCP. If not then you can make a group and make the group the owner of the directory. 

 

In order to make them able to just SCP you would run which in turn would make expedition the owner instead of the www-data which is used to process them 

sudo chown -R expedition:expedition PALogs/ 

L1 Bithead

Does anyone have any suggestions? I'm still unable to process the firewall logs and would really like to get this working.

Thanks!

send me an email to fwmigrate at paloaltonetworks dot com and we can do a Zoom session

 

If you update to the latest version of Expedition 1.0.98, you will get more information regarding the logs that you are attempting to process.

 

Things that may occur:

  1. The log is incorrectly formated
    • This log won't be able to be pre-processed
  2. The log is not supported (PANOS version below 7.1)
    • This log won't be able to be pre-processed
  3. The path for MLearning (Main Screen -> Settings -> M.Learning -> Temporary data structure folder PATH) is not a valid path or www-data does not have rights to reaching it and writing inside. Also, this should have been alerted in them main Dashboard HeathChecks
    • We can't pre-process any logs, as we can't write into a folder to store the pre-processing results
  4. The logs are way too big for the resources we have provided to Expedition. We should see an error in the file /tmp/error_logCoCo
    • We need to increase the Expedition resources, by adding more RAM and/or CPUs and deleting the file located at /home/userSpace/environmentParameters.php. (This may be optimized in the future to assist on the correct Expedition's resource tunning)
  5. The GUI offers you a link to http://<ip>:4050 and this link directs us to a web site that shows some task progress. 
    • All is fine. We need to let it continue until it completes, or fails as in case 4

 

 

 

How does on upgrade to the latest version?  I've tried the  apt-get update and apt-get expedition beta cmd's and I keep getting the "1.0.8x " is the latest version.  I've tried the unauthenticated -o commands as well, and still can't get the OS to recognize that the install is old.  Any suggestions on upgrading/where to find the latest hotfixes/installs for this?

  • 24068 Views
  • 51 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!