Expedition csv logs stuck in pending

[joeyjoejoe]

Hi everyone,

I have added firewall logs from our Palo Alto 5000 series to the Expedition VM /PALogs . I have copied the orginal .csv as a duplicate with root as the owner and the original with expedition as the owner. Both files appear in Devices > M.LEARNING. When I run Process Files the job remains in pending and nothing happens. Any ideas what the issue may be?

[rschunk]

The file a sent directly by the firewall and this is the serial number configured in Expedition.  Panorama is not involved.

To get SCP to work I nee dto change oswer to expedition.  Files then transfer without an issue but the files never show up under M.Learning to process via the Web interface (files are on Expedition and visiable via SSH shell).

Rich

[dgildelaig]

If you are certain that the serial number is correct, then I would suggest to check the following.

1. The file is readbale by www-data.
   www-data needs to have rights to read the file (does not need to own it) and it should have rights to reach the enclosing folder where the file is located. This means, if the file is in /my/path/last_day.csv, www-data should have rights to get into /my/path. Verify that this is the case.
   This is why I normally suggest to place the logs into /PALogs, as it is becomes easy to see that expedition has writing rights into the folder, and www-data has rights to access it and read the files inside.
2. The provided path is correct.
   Following the example above with a file in /my/path/last_day.csv, make sure that you provide the path to search with, for instance
   /my/path/*
   Make sure there is no spaces in the given path, and once you have seen that the path is correct, do not forget to click on the "Save" button, so Expedition remembers the path for future checks.
   
3. The log has content.
   If the logs sent by the firewall do not have any content, Expedition can't verify that the file actually belongs to the firewall with the given serial number.
   When could this happen?
   If you have a couple of FW in HA, you may have switch from primary to secondary without being aware of it. In that case, the primary FW (assuming it is the one that was configured to send the logs) is sending empty traffic logs, as it is the secondary the one processing traffic. 
   Make sure to set up both primary and secondary FW to send the traffic logs to Expedition (we suggest to send the logs to the same folder), and make use of the HA serial field to provide the serial number of the secondary FW. We can handle both FW data if their serials are provided.
4. The logs have a csv or gz extension.
   Expedition is capable of processing the traffic logs when stored in coma-separated-values (by default) or even if the files have been compressed (gz would reduce the size of the file to a 10% of the original size). However, if you 7z the file or change the extension, we won't consider the file for processing.

I hope some of those points help.

[rschunk]

There is only a single firewall involved.  SCP from the firewall does not work unless go into the CLI and change the /PALogs owner in Expedition to expedition.

Right now under Settings the Temporary Data Structure Folder is set to /opt/ml (this is the ova install default).  Do I need to change this to /PALogs files show up with thr web interface?

Thank you,  Rich

[dgildelaig]

The Temporary Data Structure Folder is used for conversion, which will come after you have managed to "find" the original CSV files.

In the main screen at Expedition, you have health checks. One of them refers to the Temporary Data Structure folder and the rights to write inside. If the check passes, then you do not need to make changes on your /opt/ml folder (unless you prefer a different folder due to space limitations).

Going back to the CSV files that can't be found, and located inside /PALogs, most probably you removed the rights for www-data to read that folder. Simply execute:

sudo chown expedition.www-data /PALogs

and later
sudo chmod -r 740 /PALogs 

This will make expedition user the owner of the folder, and www-data group (which contains www-data user) the group owner of the folder. After, www-data group will have read rights into the folder, and expedition will have write-read-execute rights. If you would prefer, you can use 770 instead of 740 to give also write rights to www-data, in order to be able to compress the files after processing or delete them (those are options when processing csv files in Expedition)

[rschunk]

Done and same issue.

expedition@pan-expedition:/PALogs$ ls -al

total 16

drwxr-----  2 expedition www-data   4096 Dec 31 11:45 .

drwxr-xr-x 24 root       root       4096 Dec 28 11:50 ..

-rw-rw-r--  1 expedition expedition  944 Dec 31 13:00 pan-panos-vm50_traffic_2018_12_31_last_calendar_day.csv

-rw-rw-r--  1 expedition expedition   17 Dec 31 12:55 ssh-export-test.txt

Dashboard is clean: no errors to remediate.  System looks good just cannot get files to show up under web interface to process.

Rich