Expedition csv logs stuck in pending

Reply
L2 Linker

Re: SCP permissions or Re: Expedition csv logs stuck in pending

The file a sent directly by the firewall and this is the serial number configured in Expedition.  Panorama is not involved.

 

To get SCP to work I nee dto change oswer to expedition.  Files then transfer without an issue but the files never show up under M.Learning to process via the Web interface (files are on Expedition and visiable via SSH shell).

 

Rich

L4 Transporter

Re: SCP permissions or Re: Expedition csv logs stuck in pending

If you are certain that the serial number is correct, then I would suggest to check the following.

  1. The file is readbale by www-data.
    www-data needs to have rights to read the file (does not need to own it) and it should have rights to reach the enclosing folder where the file is located. This means, if the file is in /my/path/last_day.csv, www-data should have rights to get into /my/path. Verify that this is the case.
    This is why I normally suggest to place the logs into /PALogs, as it is becomes easy to see that expedition has writing rights into the folder, and www-data has rights to access it and read the files inside.
  2. The provided path is correct.
    Following the example above with a file in /my/path/last_day.csv, make sure that you provide the path to search with, for instance
    /my/path/*
    Make sure there is no spaces in the given path, and once you have seen that the path is correct, do not forget to click on the "Save" button, so Expedition remembers the path for future checks.
  3. The log has content.
    If the logs sent by the firewall do not have any content, Expedition can't verify that the file actually belongs to the firewall with the given serial number.
    When could this happen?
    If you have a couple of FW in HA, you may have switch from primary to secondary without being aware of it. In that case, the primary FW (assuming it is the one that was configured to send the logs) is sending empty traffic logs, as it is the secondary the one processing traffic. 
    Make sure to set up both primary and secondary FW to send the traffic logs to Expedition (we suggest to send the logs to the same folder), and make use of the HA serial field to provide the serial number of the secondary FW. We can handle both FW data if their serials are provided.
  4. The logs have a csv or gz extension.
    Expedition is capable of processing the traffic logs when stored in coma-separated-values (by default) or even if the files have been compressed (gz would reduce the size of the file to a 10% of the original size). However, if you 7z the file or change the extension, we won't consider the file for processing.

 

I hope some of those points help.

L2 Linker

Re: SCP permissions or Re: Expedition csv logs stuck in pending

There is only a single firewall involved.  SCP from the firewall does not work unless go into the CLI and change the /PALogs owner in Expedition to expedition.

 

Right now under Settings the Temporary Data Structure Folder is set to /opt/ml (this is the ova install default).  Do I need to change this to /PALogs files show up with thr web interface?

 

Thank you,  Rich

L4 Transporter

Re: SCP permissions or Re: Expedition csv logs stuck in pending

The Temporary Data Structure Folder is used for conversion, which will come after you have managed to "find" the original CSV files.

In the main screen at Expedition, you have health checks. One of them refers to the Temporary Data Structure folder and the rights to write inside. If the check passes, then you do not need to make changes on your /opt/ml folder (unless you prefer a different folder due to space limitations).

 

Going back to the CSV files that can't be found, and located inside /PALogs, most probably you removed the rights for www-data to read that folder. Simply execute:

sudo chown expedition.www-data /PALogs

and later
sudo chmod -r 740 /PALogs 

This will make expedition user the owner of the folder, and www-data group (which contains www-data user) the group owner of the folder. After, www-data group will have read rights into the folder, and expedition will have write-read-execute rights. If you would prefer, you can use 770 instead of 740 to give also write rights to www-data, in order to be able to compress the files after processing or delete them (those are options when processing csv files in Expedition)

L2 Linker

Re: SCP permissions or Re: Expedition csv logs stuck in pending

Done and same issue.

 

expedition@pan-expedition:/PALogs$ ls -al

total 16

drwxr-----  2 expedition www-data   4096 Dec 31 11:45 .

drwxr-xr-x 24 root       root       4096 Dec 28 11:50 ..

-rw-rw-r--  1 expedition expedition  944 Dec 31 13:00 pan-panos-vm50_traffic_2018_12_31_last_calendar_day.csv

-rw-rw-r--  1 expedition expedition   17 Dec 31 12:55 ssh-export-test.txt

 

Dashboard is clean: no errors to remediate.  System looks good just cannot get files to show up under web interface to process.

 

Rich

L4 Transporter

Re: SCP permissions or Re: Expedition csv logs stuck in pending

TL;DR

One more thing, in the ML Settings, make sure the provided IP is the correct one.

 

Long Explanation

Even it may sound strange, we desinged Expedition to allow being split into two parts, the config management part, and the Machine Learning part. 
In most cases (maybe 99%), both parts are the same machine. However, the management part needs to know how to reach the Machine Learning part for finding the CSV logs, converting them into parquets, performing data analytics to generate rules, etc.

 

Why do we desgined it this way? We had on mind that some users may require a very performing unit for data analytics, for instance with 24 CPU and 256GB of RAM. Maybe, they even have a cluster for processing Spark jobs (whic we use for Machine Learning). In that case, we started the desing to untangle Expedition into a heavy part (that could be shared with other projects and perform the data analytics) and a light part that handles configurations and rest of Expedition features.

 

 

L2 Linker

Re: SCP permissions or Re: Expedition csv logs stuck in pending

it is- 192.168.55.120 (default IP address for LITB Expedition).  Rich

Highlighted
L4 Transporter

Re: SCP permissions or Re: Expedition csv logs stuck in pending

If it still does not resolve the issue, please send us an email to fwmigrate at paloaltonetworks dot com, and we may try to do a live session to help you.

L1 Bithead

Re: Expedition csv logs stuck in pending

I am still having the same issue, and the result is also blank.
L3 Networker

Re: Expedition csv logs stuck in pending

you didn't by chance fill up your drive on the expedition vm did you?  

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!