Won't process CSV files

Reply
Highlighted
L2 Linker

Won't process CSV files

I blew away my VM and reloaded it with an OVA our PA SE created for us.  It installed and functioned just like the one I had created and tried the first half of the week, but I wanted to start with a clean slate.  I'm using the specs from the Workstation image of 1 cpu, 1.5Gb RAM, 40Gb disk.  v1.0.84

 

I did the following:

Created the M.learning /data directory and used chown to set permissions for www-data

Created the /logs directory and set permissions with chmod 777

SCP'd a file from the firewall to Expedition (138MB)

Created the Device

Created API key, saved

Retrieved Contents, saved

Defined m.learning directory to search, saved

Checked the box on the csv to  process

Clicked process.  Button changed color, nothing happens.  Still says Ready.

Settings, Jobs only shows retrieving the contents of the Device.

 

This is the same issue I had on the last image I used.

L4 Transporter

Re: Won't process CSV files

Could you verify that you do not have warnings in the main Dashboard HealthChecks?

 

I could not replicate the issue you describe but some of the following could be the source:

- Either we did not find logs to process in the path you provided

- The files are not having a valid/expected CSV format

- The files do not belong to the firewall we have defined (serial does not match)

- All the files are ignored (with the red icon)

- The /data folder is not actually writable by www-data

L2 Linker

Re: Won't process CSV files

expedition@Expedition:/$ ls -al
drwxr-xr-x 2 www-data www-data 4096 Jun 27 16:05 datastore
drwxrwxrwx 2 root root 4096 Jun 27 16:20 logs

 

I copied the name, serial, and IP directly from the Dashboard of the FW and the FW SCP'd the log to Expedition.

exp2.PNGexp1.PNG

L2 Linker

Re: Won't process CSV files

expedition@Expedition:/$ cd /logs
expedition@Expedition:/logs$ ls -la
-rw-rw-r-- 1 expedition expedition 143755393 Jun 27 16:23 NMELBPPAFW01_traffic_2018_06_28_last_calendar_day.csv

L4 Transporter

Re: Won't process CSV files

I noticed you talk about /data folder, but it seems that Expedition will try to use /datastore.

Also, could you check that your Expedition is updated and try again?
L2 Linker

Re: Won't process CSV files

Correct.  /datastore

 

Are my permissions correct for the two folders?

expedition@Expedition:/$ ls -al
drwxr-xr-x 2 www-data www-data 4096 Jun 27 16:05 datastore
drwxrwxrwx 2 root root 4096 Jun 27 16:20 logs

 

I successfully upgraded to 1.0.94 and now when I tell it to process the file,  it says that there are no files to process.  I did delete yesterdays and had the firewall SCP a new one.

exp4.PNG

L4 Transporter

Re: Won't process CSV files

It looks that you have rights to read files in the folder, and also to create the parquets.

I assume that those two folders are hanging from the root folder ( / ).

Send me your contact to fwmigrate at paloaltonetworks dot com and let's try to have a zoom session tomorrow.
We will post the resolution back here for others with similar problems.
L2 Linker

Re: Won't process CSV files

They are under Expedition, so /home/expedition/logs and /home/expedition/datastore.

 

When I change the CSV search to /home/expedition/logs it doesn't see anything.

 

I sent you my information.

L2 Linker

Re: Won't process CSV files

I added another firewall and had it send it's log via SCP and it processed it fine. Success!

I deleted the original device that existed before the 1.0.94 upgrade and recreated it. Same result. "No files to process"

I deleted the device again and deleted the config folder in /home/userSpace/devices and then recreated the device. Same result. "No files to process".

 

If it matters, it is a PA-500 running 5.0.9 code.

Export the logs from Panorama that is running newer code instead?

Can Expedition pull the logs directly from Panorama using the log connector?

I've read every other thread in this Discussion trying to figure out what options there are and how to do them.  Thanks!

 

L4 Transporter

Re: Won't process CSV files

The CSV logs formats supported are from 7.1 onwards.

 

Most probably the format for 5.0 does not comply with the formats we currently support.

 

We aim at giving log support to the supported versions of PANOS. PANOS 6.1 is still supported (until October, if I remember correctly).

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!