forwarded logs not deleting after processing

[BenKnorr2]

I have Panorama configured as a device in Expedition. Devices managed by Panorama have been imported/retrieved into the device within Expedition. Some stuff I've done/is configured:

• crontab is set to fix permissions on imported logs daily at midnight. It runs successfully and resulting files look like they have the right permissions:
  -rw-rw---- 1 expedition www-data 184G Sep 17 17:56 PA5220_traffic.....
• My daily scheduled log processing is set for 4AM
• The M.Learning component in the device (Panorama) is set to "auto process CSV log files" and appears to do this. I've been able to analyze rules in a project using this info.
• I have "after process: Delete" configured, but it doesn't appear to work

I've also got another thread out there regarding the "process Enabled Files" option that is greyed out in this context. The only way I can process these logs is by letting the daily processing schedule catch up to them, or manually changing that schedule to be 2 min from now, for instance.

In any case, the server quickly fills up with space as logs aren't being deleted after processing. My thinking is that logs are uploaded at 1600, ACL changed at 0000, then auto processing kicking off at 0400. So far it seems to all work except the deleting part. Any tips?

[lychiang]

Hi @BenKnorr2 , this might be permission issue , due to the www-data is not able to delete the files, assume your log stored in /PALogs folder, please do the following:

Looks like form your screenshot, it already showing the correct owner and group.  If  it's not correct , just do the below command to change it: 

sudo chown expedition.www-data /PALogs

and later
sudo chmod -r 770 /PALogs 

This will make expedition user the owner of the folder, and www-data group (which contains www-data user) the group owner of the folder. After, www-data group will have readwrite rights into the folder, and expedition will have write-read-execute rights. 770 give write rights to www-data, in order to be able to compress the files after processing or delete them (those are options when processing csv files in Expedition)

[BenKnorr2]

in my case, /PAlogs is where the parquet files are stored. /home/expedition/logs is where I've got my FW exporting logs to. FWIW, I have this exact same setup in my lab and everything is the same except I have a FW configured as a device instead of Panorama.

I'm not looking to delete parquet files, just the massive exported traffic logs from my FW.

[lychiang]

Then you will verify the permission and owner for the folder /home/expedition/logs

[BenKnorr2]

Sorry, wasn't clear earlier. I put the ACL fix script shown in Settings/M.Learning in the "CSV log file rights" section. It lists slightly erroneous (one extra *) but otherwise very helpful line to put into crontab to have a built-in Expedition script fix ACLs on filesystem so logs can be deleted after processing. This part does work. An unprocessed imported log file that has expedition:expedition ownership changes to 660 expedition:www-data after running the script. This part has been consistent.