forwarded logs not deleting after processing

Showing results for 
Show  only  | Search instead for 
Did you mean: 

forwarded logs not deleting after processing

L2 Linker

I have Panorama configured as a device in Expedition. Devices managed by Panorama have been imported/retrieved into the device within Expedition. Some stuff I've done/is configured:

  • crontab is set to fix permissions on imported logs daily at midnight. It runs successfully and resulting files look like they have the right permissions:
    -rw-rw---- 1 expedition www-data 184G Sep 17 17:56 PA5220_traffic.....
  • My daily scheduled log processing is set for 4AM
  • The M.Learning component in the device (Panorama) is set to "auto process CSV log files" and appears to do this. I've been able to analyze rules in a project using this info.
  • I have "after process: Delete" configured, but it doesn't appear to work

I've also got another thread out there regarding the "process Enabled Files" option that is greyed out in this context. The only way I can process these logs is by letting the daily processing schedule catch up to them, or manually changing that schedule to be 2 min from now, for instance.


In any case, the server quickly fills up with space as logs aren't being deleted after processing. My thinking is that logs are uploaded at 1600, ACL changed at 0000, then auto processing kicking off at 0400. So far it seems to all work except the deleting part. Any tips?


Accepted Solutions

For Panorama managed FW, you will need to check the FW ML setting, go to the "Device" tab, click on the "show all devices" icon as on the right upper corner as shown in the below screenshot, and find the FW that's matching your traffic logs, check the ML setting on that firewall.  



View solution in original post


L6 Presenter

Hi @BenKnorr2 , this might be permission issue , due to the www-data is not able to delete the files, assume your log stored in /PALogs folder, please do the following:

Looks like form your screenshot, it already showing the correct owner and group.  If  it's not correct , just do the below command to change it: 

sudo chown expedition.www-data /PALogs

and later
sudo chmod -r 770 /PALogs 

This will make expedition user the owner of the folder, and www-data group (which contains www-data user) the group owner of the folder. After, www-data group will have readwrite rights into the folder, and expedition will have write-read-execute rights. 770 give write rights to www-data, in order to be able to compress the files after processing or delete them (those are options when processing csv files in Expedition)

in my case, /PAlogs is where the parquet files are stored. /home/expedition/logs is where I've got my FW exporting logs to. FWIW, I have this exact same setup in my lab and everything is the same except I have a FW configured as a device instead of Panorama.

I'm not looking to delete parquet files, just the massive exported traffic logs from my FW.

Then you will verify the permission and owner for the folder /home/expedition/logs

Sorry, wasn't clear earlier. I put the ACL fix script shown in Settings/M.Learning in the "CSV log file rights" section. It lists slightly erroneous (one extra *) but otherwise very helpful line to put into crontab to have a built-in Expedition script fix ACLs on filesystem so logs can be deleted after processing. This part does work. An unprocessed imported log file that has expedition:expedition ownership changes to 660 expedition:www-data after running the script. This part has been consistent.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!