Won't process CSV files

[Esfeld]

I blew away my VM and reloaded it with an OVA our PA SE created for us.  It installed and functioned just like the one I had created and tried the first half of the week, but I wanted to start with a clean slate.  I'm using the specs from the Workstation image of 1 cpu, 1.5Gb RAM, 40Gb disk.  v1.0.84

I did the following:

Created the M.learning /data directory and used chown to set permissions for www-data

Created the /logs directory and set permissions with chmod 777

SCP'd a file from the firewall to Expedition (138MB)

Created the Device

Created API key, saved

Retrieved Contents, saved

Defined m.learning directory to search, saved

Checked the box on the csv to  process

Clicked process.  Button changed color, nothing happens.  Still says Ready.

Settings, Jobs only shows retrieving the contents of the Device.

This is the same issue I had on the last image I used.

[dgildelaig]

Could you verify that you do not have warnings in the main Dashboard HealthChecks?

I could not replicate the issue you describe but some of the following could be the source:

- Either we did not find logs to process in the path you provided

- The files are not having a valid/expected CSV format

- The files do not belong to the firewall we have defined (serial does not match)

- All the files are ignored (with the red icon)

- The /data folder is not actually writable by www-data

[Esfeld]

expedition@Expedition:/$ ls -al
drwxr-xr-x 2 www-data www-data 4096 Jun 27 16:05 datastore
drwxrwxrwx 2 root root 4096 Jun 27 16:20 logs

I copied the name, serial, and IP directly from the Dashboard of the FW and the FW SCP'd the log to Expedition.

[Esfeld]

expedition@Expedition:/$ cd /logs
expedition@Expedition:/logs$ ls -la
-rw-rw-r-- 1 expedition expedition 143755393 Jun 27 16:23 NMELBPPAFW01_traffic_2018_06_28_last_calendar_day.csv

[dgildelaig]

I noticed you talk about /data folder, but it seems that Expedition will try to use /datastore.

Also, could you check that your Expedition is updated and try again?

[Esfeld]

Correct.  /datastore

Are my permissions correct for the two folders?

expedition@Expedition:/$ ls -al
drwxr-xr-x 2 www-data www-data 4096 Jun 27 16:05 datastore
drwxrwxrwx 2 root root 4096 Jun 27 16:20 logs

I successfully upgraded to 1.0.94 and now when I tell it to process the file,  it says that there are no files to process.  I did delete yesterdays and had the firewall SCP a new one.

[dgildelaig]

It looks that you have rights to read files in the folder, and also to create the parquets.

I assume that those two folders are hanging from the root folder ( / ).

Send me your contact to fwmigrate at paloaltonetworks dot com and let's try to have a zoom session tomorrow.
We will post the resolution back here for others with similar problems.

[Esfeld]

They are under Expedition, so /home/expedition/logs and /home/expedition/datastore.

When I change the CSV search to /home/expedition/logs it doesn't see anything.

I sent you my information.

[Esfeld]

I added another firewall and had it send it's log via SCP and it processed it fine. Success!

I deleted the original device that existed before the 1.0.94 upgrade and recreated it. Same result. "No files to process"

I deleted the device again and deleted the config folder in /home/userSpace/devices and then recreated the device. Same result. "No files to process".

If it matters, it is a PA-500 running 5.0.9 code.

Export the logs from Panorama that is running newer code instead?

Can Expedition pull the logs directly from Panorama using the log connector?

I've read every other thread in this Discussion trying to figure out what options there are and how to do them.  Thanks!

[dgildelaig]

The CSV logs formats supported are from 7.1 onwards.

Most probably the format for 5.0 does not comply with the formats we currently support.

We aim at giving log support to the supported versions of PANOS. PANOS 6.1 is still supported (until October, if I remember correctly).