forwarded logs not deleting after processing

Announcements

Changes to the LIVEcommunity experience are coming soon... Here's what you need to know.

Reply
BenKnorr2
L2 Linker

forwarded logs not deleting after processing

I have Panorama configured as a device in Expedition. Devices managed by Panorama have been imported/retrieved into the device within Expedition. Some stuff I've done/is configured:

  • crontab is set to fix permissions on imported logs daily at midnight. It runs successfully and resulting files look like they have the right permissions:
    -rw-rw---- 1 expedition www-data 184G Sep 17 17:56 PA5220_traffic.....
  • My daily scheduled log processing is set for 4AM
  • The M.Learning component in the device (Panorama) is set to "auto process CSV log files" and appears to do this. I've been able to analyze rules in a project using this info.
  • I have "after process: Delete" configured, but it doesn't appear to work

I've also got another thread out there regarding the "process Enabled Files" option that is greyed out in this context. The only way I can process these logs is by letting the daily processing schedule catch up to them, or manually changing that schedule to be 2 min from now, for instance.

 

In any case, the server quickly fills up with space as logs aren't being deleted after processing. My thinking is that logs are uploaded at 1600, ACL changed at 0000, then auto processing kicking off at 0400. So far it seems to all work except the deleting part. Any tips?


Accepted Solutions
lychiang
L4 Transporter

For Panorama managed FW, you will need to check the FW ML setting, go to the "Device" tab, click on the "show all devices" icon as on the right upper corner as shown in the below screenshot, and find the FW that's matching your traffic logs, check the ML setting on that firewall.  

 

Save.png

View solution in original post


All Replies
lychiang
L4 Transporter

Hi @BenKnorr2 , this might be permission issue , due to the www-data is not able to delete the files, assume your log stored in /PALogs folder, please do the following:

Looks like form your screenshot, it already showing the correct owner and group.  If  it's not correct , just do the below command to change it: 

sudo chown expedition.www-data /PALogs

and later
sudo chmod -r 770 /PALogs 

This will make expedition user the owner of the folder, and www-data group (which contains www-data user) the group owner of the folder. After, www-data group will have readwrite rights into the folder, and expedition will have write-read-execute rights. 770 give write rights to www-data, in order to be able to compress the files after processing or delete them (those are options when processing csv files in Expedition)

BenKnorr2
L2 Linker

in my case, /PAlogs is where the parquet files are stored. /home/expedition/logs is where I've got my FW exporting logs to. FWIW, I have this exact same setup in my lab and everything is the same except I have a FW configured as a device instead of Panorama.

I'm not looking to delete parquet files, just the massive exported traffic logs from my FW.

lychiang
L4 Transporter

Then you will verify the permission and owner for the folder /home/expedition/logs

BenKnorr2
L2 Linker

Sorry, wasn't clear earlier. I put the ACL fix script shown in Settings/M.Learning in the "CSV log file rights" section. It lists slightly erroneous (one extra *) but otherwise very helpful line to put into crontab to have a built-in Expedition script fix ACLs on filesystem so logs can be deleted after processing. This part does work. An unprocessed imported log file that has expedition:expedition ownership changes to 660 expedition:www-data after running the script. This part has been consistent.

BenKnorr2
L2 Linker

Is this a Panorama thing? The permissions on these files look correct, but the auto processing phase never deletes them.

lychiang
L4 Transporter

@BenKnorr2 Have you tried the below commands already?

 

sudo chown expedition.www-data/home/expedition/logs

and later
sudo chmod -r 770/home/expedition/logs

dgildelaig
L5 Sessionator

Have you verified that the device that reported those logs has the After Process action set to delete?

Maybe you had it for the Panorama, but the device says something else, and we will take action based on what the device states.

 

BenKnorr2
L2 Linker

Thanks for the response. Unfortunately, no, permissions have been consistent and what appears to be correct since beginning. In Expedition GUI / Settings /M.Learning, there is a blurb at the bottom for CSV and file rights:

Scheduled Log Export from FW devices may export your log files as expedition owned files.
In case you want to activate delete after processing the CSV logs, make sure that www-data has write rights over the files.
You can achieve it by adding the following into your root cron (the following example will verify the file rights every day at 00:05 am):
00 05 * * * * php /var/www/html/OS/spark/scripts/changeCSVLogRights.php

I entered this in my crontab (syntax shown above looks incorrect though; has an extra *) and it fixes permissions for 660 / chown www-data:expedition for any logs in a project that are set to be autoprocessed but haven't been processed yet. This does work. This step helped in my lab environment to get autoprocessing+deleteafter to succeed.

 

For what it's worth, here are two log files: one with 770 was set immediately after it was uploaded to expedition prior to processing. The other was set using the built-in script that runs at 00:05 nightly via cron. Both files have since been "auto processed" with "delete after processing" checked, but no deletion occurred after processing. Just showing that 770/660 seem to have no impact on problem.

expedition@Expedition:~/logs$ ls -lah
total 317G
drwxrwxr-x 2 expedition expedition 4.0K Sep 28 15:56 .
drwxr-xr-x 5 expedition expedition 4.0K Sep 10 08:41 ..
-rwxrwx--- 1 expedition www-data 157G Sep 27 17:39 PA5250-PRI_traffic_2020_09_27_last_calendar_day.csv
-rw-rw---- 1 expedition www-data 161G Sep 28 17:50 PA5250-PRI_traffic_2020_09_28_last_calendar_day.csv

My lab environment has a FW sending logs via SCP to expedition, and the device in expedition is for the firewall itself w/no panorama. "process enabled files" button in device tab works for manual processing (have another thread on this board about this part). Logs weren't getting deleted until I put the php script in crontab, now it works good.  My prod environment has FW sending logs via SCP to expedition, but the device is panorama with managed devices/config imported to it. "process enabled files" is greyed out and has never been usable, for what its worth, and deleting logs after auto processing has never worked.

lychiang
L4 Transporter

For Panorama managed FW, you will need to check the FW ML setting, go to the "Device" tab, click on the "show all devices" icon as on the right upper corner as shown in the below screenshot, and find the FW that's matching your traffic logs, check the ML setting on that firewall.  

 

Save.png

View solution in original post

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!