topic Re: forwarded logs not deleting after processing in Expedition Discussions

forwarded logs not deleting after processing

BenKnorr2 — Fri, 18 Sep 2020 16:05:25 GMT

I have Panorama configured as a device in Expedition. Devices managed by Panorama have been imported/retrieved into the device within Expedition. Some stuff I've done/is configured:

crontab is set to fix permissions on imported logs daily at midnight. It runs successfully and resulting files look like they have the right permissions:
-rw-rw---- 1 expedition www-data 184G Sep 17 17:56 PA5220_traffic.....
My daily scheduled log processing is set for 4AM
The M.Learning component in the device (Panorama) is set to "auto process CSV log files" and appears to do this. I've been able to analyze rules in a project using this info.
I have "after process: Delete" configured, but it doesn't appear to work

I've also got another thread out there regarding the "process Enabled Files" option that is greyed out in this context. The only way I can process these logs is by letting the daily processing schedule catch up to them, or manually changing that schedule to be 2 min from now, for instance.

In any case, the server quickly fills up with space as logs aren't being deleted after processing. My thinking is that logs are uploaded at 1600, ACL changed at 0000, then auto processing kicking off at 0400. So far it seems to all work except the deleting part. Any tips?

Re: forwarded logs not deleting after processing

lychiang — Fri, 18 Sep 2020 16:40:24 GMT

Hi @BenKnorr2 , this might be permission issue , due to the www-data is not able to delete the files, assume your log stored in /PALogs folder, please do the following:

Looks like form your screenshot, it already showing the correct owner and group. If it's not correct , just do the below command to change it:

sudo chown expedition.www-data /PALogs

and later
sudo chmod -r 770 /PALogs

This will make expedition user the owner of the folder, and www-data group (which contains www-data user) the group owner of the folder. After, www-data group will have readwrite rights into the folder, and expedition will have write-read-execute rights. 770 give write rights to www-data, in order to be able to compress the files after processing or delete them (those are options when processing csv files in Expedition)

Re: forwarded logs not deleting after processing

BenKnorr2 — Fri, 18 Sep 2020 16:41:47 GMT

in my case, /PAlogs is where the parquet files are stored. /home/expedition/logs is where I've got my FW exporting logs to. FWIW, I have this exact same setup in my lab and everything is the same except I have a FW configured as a device instead of Panorama.

I'm not looking to delete parquet files, just the massive exported traffic logs from my FW.

Re: forwarded logs not deleting after processing

lychiang — Fri, 18 Sep 2020 16:45:03 GMT

Then you will verify the permission and owner for the folder /home/expedition/logs

Re: forwarded logs not deleting after processing

BenKnorr2 — Fri, 18 Sep 2020 16:54:41 GMT

Sorry, wasn't clear earlier. I put the ACL fix script shown in Settings/M.Learning in the "CSV log file rights" section. It lists slightly erroneous (one extra *) but otherwise very helpful line to put into crontab to have a built-in Expedition script fix ACLs on filesystem so logs can be deleted after processing. This part does work. An unprocessed imported log file that has expedition:expedition ownership changes to 660 expedition:www-data after running the script. This part has been consistent.

Re: forwarded logs not deleting after processing

BenKnorr2 — Mon, 28 Sep 2020 15:47:04 GMT

Is this a Panorama thing? The permissions on these files look correct, but the auto processing phase never deletes them.

Re: forwarded logs not deleting after processing

lychiang — Mon, 28 Sep 2020 15:50:38 GMT

@BenKnorr2 Have you tried the below commands already?

sudo chown expedition.www-data/home/expedition/logs

and later
sudo chmod -r 770/home/expedition/logs

Re: forwarded logs not deleting after processing

dgildelaig — Mon, 28 Sep 2020 16:41:45 GMT

Have you verified that the device that reported those logs has the After Process action set to delete?

Maybe you had it for the Panorama, but the device says something else, and we will take action based on what the device states.

Re: forwarded logs not deleting after processing

BenKnorr2 — Tue, 29 Sep 2020 18:32:49 GMT

Thanks for the response. Unfortunately, no, permissions have been consistent and what appears to be correct since beginning. In Expedition GUI / Settings /M.Learning, there is a blurb at the bottom for CSV and file rights:

Scheduled Log Export from FW devices may export your log files as expedition owned files.
In case you want to activate delete after processing the CSV logs, make sure that www-data has write rights over the files.
You can achieve it by adding the following into your root cron (the following example will verify the file rights every day at 00:05 am):
00 05 * * * * php /var/www/html/OS/spark/scripts/changeCSVLogRights.php

I entered this in my crontab (syntax shown above looks incorrect though; has an extra *) and it fixes permissions for 660 / chown www-data:expedition for any logs in a project that are set to be autoprocessed but haven't been processed yet. This does work. This step helped in my lab environment to get autoprocessing+deleteafter to succeed.

For what it's worth, here are two log files: one with 770 was set immediately after it was uploaded to expedition prior to processing. The other was set using the built-in script that runs at 00:05 nightly via cron. Both files have since been "auto processed" with "delete after processing" checked, but no deletion occurred after processing. Just showing that 770/660 seem to have no impact on problem.

expedition@Expedition:~/logs$ ls -lah
total 317G
drwxrwxr-x 2 expedition expedition 4.0K Sep 28 15:56 .
drwxr-xr-x 5 expedition expedition 4.0K Sep 10 08:41 ..
-rwxrwx--- 1 expedition www-data 157G Sep 27 17:39 PA5250-PRI_traffic_2020_09_27_last_calendar_day.csv
-rw-rw---- 1 expedition www-data 161G Sep 28 17:50 PA5250-PRI_traffic_2020_09_28_last_calendar_day.csv

My lab environment has a FW sending logs via SCP to expedition, and the device in expedition is for the firewall itself w/no panorama. "process enabled files" button in device tab works for manual processing (have another thread on this board about this part). Logs weren't getting deleted until I put the php script in crontab, now it works good. My prod environment has FW sending logs via SCP to expedition, but the device is panorama with managed devices/config imported to it. "process enabled files" is greyed out and has never been usable, for what its worth, and deleting logs after auto processing has never worked.

Re: forwarded logs not deleting after processing

lychiang — Tue, 29 Sep 2020 18:47:37 GMT

For Panorama managed FW, you will need to check the FW ML setting, go to the "Device" tab, click on the "show all devices" icon as on the right upper corner as shown in the below screenshot, and find the FW that's matching your traffic logs, check the ML setting on that firewall.

Re: forwarded logs not deleting after processing

BenKnorr2 — Tue, 29 Sep 2020 20:45:05 GMT

I only have Panorama set in devices, but the managed firewalls have been retrieved within it. Goal is to take rules from panorama device groups and use ML on traffic.

There is no place to set "after processing" action for the firewall themselves in expedition when panorama is the device in question. Am I missing something there and this isn't supported in the first place?

Re: forwarded logs not deleting after processing

lychiang — Tue, 29 Sep 2020 20:48:21 GMT

Even you received the traffic log form Panorama, The ML setting you to need to check is on the FW device not on Panorama, you will make sure the ML setting is set to delete the file after processing.