topic Re: Won't process CSV files in Expedition Discussions

Won't process CSV files

Esfeld — Wed, 27 Jun 2018 21:48:20 GMT

I blew away my VM and reloaded it with an OVA our PA SE created for us. It installed and functioned just like the one I had created and tried the first half of the week, but I wanted to start with a clean slate. I'm using the specs from the Workstation image of 1 cpu, 1.5Gb RAM, 40Gb disk. v1.0.84

I did the following:

Created the M.learning /data directory and used chown to set permissions for www-data

Created the /logs directory and set permissions with chmod 777

SCP'd a file from the firewall to Expedition (138MB)

Created the Device

Created API key, saved

Retrieved Contents, saved

Defined m.learning directory to search, saved

Checked the box on the csv to process

Clicked process. Button changed color, nothing happens. Still says Ready.

Settings, Jobs only shows retrieving the contents of the Device.

This is the same issue I had on the last image I used.

Re: Won't process CSV files

dgildelaig — Thu, 28 Jun 2018 10:17:10 GMT

Could you verify that you do not have warnings in the main Dashboard HealthChecks?

I could not replicate the issue you describe but some of the following could be the source:

- Either we did not find logs to process in the path you provided

- The files are not having a valid/expected CSV format

- The files do not belong to the firewall we have defined (serial does not match)

- All the files are ignored (with the red icon)

- The /data folder is not actually writable by www-data

Re: Won't process CSV files

Esfeld — Thu, 28 Jun 2018 15:30:48 GMT

expedition@Expedition:/$ ls -al
drwxr-xr-x 2 www-data www-data 4096 Jun 27 16:05 datastore
drwxrwxrwx 2 root root 4096 Jun 27 16:20 logs

I copied the name, serial, and IP directly from the Dashboard of the FW and the FW SCP'd the log to Expedition.

Re: Won't process CSV files

Esfeld — Thu, 28 Jun 2018 15:36:05 GMT

expedition@Expedition:/$ cd /logs
expedition@Expedition:/logs$ ls -la
-rw-rw-r-- 1 expedition expedition 143755393 Jun 27 16:23 NMELBPPAFW01_traffic_2018_06_28_last_calendar_day.csv

Re: Won't process CSV files

dgildelaig — Thu, 28 Jun 2018 15:48:33 GMT

I noticed you talk about /data folder, but it seems that Expedition will try to use /datastore.

Also, could you check that your Expedition is updated and try again?

Re: Won't process CSV files

Esfeld — Thu, 28 Jun 2018 20:19:51 GMT

Correct. /datastore

Are my permissions correct for the two folders?

expedition@Expedition:/$ ls -al
drwxr-xr-x 2 www-data www-data 4096 Jun 27 16:05 datastore
drwxrwxrwx 2 root root 4096 Jun 27 16:20 logs

I successfully upgraded to 1.0.94 and now when I tell it to process the file, it says that there are no files to process. I did delete yesterdays and had the firewall SCP a new one.

Re: Won't process CSV files

dgildelaig — Thu, 28 Jun 2018 20:29:17 GMT

It looks that you have rights to read files in the folder, and also to create the parquets.

I assume that those two folders are hanging from the root folder ( / ).

Send me your contact to fwmigrate at paloaltonetworks dot com and let's try to have a zoom session tomorrow.
We will post the resolution back here for others with similar problems.

Re: Won't process CSV files

Esfeld — Thu, 28 Jun 2018 20:40:31 GMT

They are under Expedition, so /home/expedition/logs and /home/expedition/datastore.

When I change the CSV search to /home/expedition/logs it doesn't see anything.

I sent you my information.

Re: Won't process CSV files

Esfeld — Thu, 28 Jun 2018 22:30:06 GMT

I added another firewall and had it send it's log via SCP and it processed it fine. Success!

I deleted the original device that existed before the 1.0.94 upgrade and recreated it. Same result. "No files to process"

I deleted the device again and deleted the config folder in /home/userSpace/devices and then recreated the device. Same result. "No files to process".

If it matters, it is a PA-500 running 5.0.9 code.

Export the logs from Panorama that is running newer code instead?

Can Expedition pull the logs directly from Panorama using the log connector?

I've read every other thread in this Discussion trying to figure out what options there are and how to do them. Thanks!

Re: Won't process CSV files

dgildelaig — Mon, 02 Jul 2018 08:24:59 GMT

The CSV logs formats supported are from 7.1 onwards.

Most probably the format for 5.0 does not comply with the formats we currently support.

We aim at giving log support to the supported versions of PANOS. PANOS 6.1 is still supported (until October, if I remember correctly).

Re: Won't process CSV files

SShnap — Thu, 02 Aug 2018 18:45:28 GMT

Try to update for the latest version, I had the same problems until I tried to update and it fixed all of them.

My current version is expedition 1.0.101

run the commands

sudo apt-get update

sudo apt-get install expedition-beta

good luck