forwarded logs not deleting after processing

[BenKnorr2]

I have Panorama configured as a device in Expedition. Devices managed by Panorama have been imported/retrieved into the device within Expedition. Some stuff I've done/is configured:

• crontab is set to fix permissions on imported logs daily at midnight. It runs successfully and resulting files look like they have the right permissions:
  -rw-rw---- 1 expedition www-data 184G Sep 17 17:56 PA5220_traffic.....
• My daily scheduled log processing is set for 4AM
• The M.Learning component in the device (Panorama) is set to "auto process CSV log files" and appears to do this. I've been able to analyze rules in a project using this info.
• I have "after process: Delete" configured, but it doesn't appear to work

I've also got another thread out there regarding the "process Enabled Files" option that is greyed out in this context. The only way I can process these logs is by letting the daily processing schedule catch up to them, or manually changing that schedule to be 2 min from now, for instance.

In any case, the server quickly fills up with space as logs aren't being deleted after processing. My thinking is that logs are uploaded at 1600, ACL changed at 0000, then auto processing kicking off at 0400. So far it seems to all work except the deleting part. Any tips?

[BenKnorr2]

Is this a Panorama thing? The permissions on these files look correct, but the auto processing phase never deletes them.

[lychiang]

@BenKnorr2 Have you tried the below commands already?

sudo chown expedition.www-data/home/expedition/logs

and later
sudo chmod -r 770/home/expedition/logs

[dgildelaig]

Have you verified that the device that reported those logs has the After Process action set to delete?

Maybe you had it for the Panorama, but the device says something else, and we will take action based on what the device states.

[BenKnorr2]

Thanks for the response. Unfortunately, no, permissions have been consistent and what appears to be correct since beginning. In Expedition GUI / Settings /M.Learning, there is a blurb at the bottom for CSV and file rights:

Scheduled Log Export from FW devices may export your log files as expedition owned files.
In case you want to activate delete after processing the CSV logs, make sure that www-data has write rights over the files.
You can achieve it by adding the following into your root cron (the following example will verify the file rights every day at 00:05 am):
00 05 * * * * php /var/www/html/OS/spark/scripts/changeCSVLogRights.php

I entered this in my crontab (syntax shown above looks incorrect though; has an extra *) and it fixes permissions for 660 / chown www-data:expedition for any logs in a project that are set to be autoprocessed but haven't been processed yet. This does work. This step helped in my lab environment to get autoprocessing+deleteafter to succeed.

For what it's worth, here are two log files: one with 770 was set immediately after it was uploaded to expedition prior to processing. The other was set using the built-in script that runs at 00:05 nightly via cron. Both files have since been "auto processed" with "delete after processing" checked, but no deletion occurred after processing. Just showing that 770/660 seem to have no impact on problem.

expedition@Expedition:~/logs$ ls -lah
total 317G
drwxrwxr-x 2 expedition expedition 4.0K Sep 28 15:56 .
drwxr-xr-x 5 expedition expedition 4.0K Sep 10 08:41 ..
-rwxrwx--- 1 expedition www-data 157G Sep 27 17:39 PA5250-PRI_traffic_2020_09_27_last_calendar_day.csv
-rw-rw---- 1 expedition www-data 161G Sep 28 17:50 PA5250-PRI_traffic_2020_09_28_last_calendar_day.csv

My lab environment has a FW sending logs via SCP to expedition, and the device in expedition is for the firewall itself w/no panorama. "process enabled files" button in device tab works for manual processing (have another thread on this board about this part). Logs weren't getting deleted until I put the php script in crontab, now it works good.  My prod environment has FW sending logs via SCP to expedition, but the device is panorama with managed devices/config imported to it. "process enabled files" is greyed out and has never been usable, for what its worth, and deleting logs after auto processing has never worked.

[lychiang · Accepted solution]

For Panorama managed FW, you will need to check the FW ML setting, go to the "Device" tab, click on the "show all devices" icon as on the right upper corner as shown in the below screenshot, and find the FW that's matching your traffic logs, check the ML setting on that firewall.