09-18-2020 09:05 AM
I have Panorama configured as a device in Expedition. Devices managed by Panorama have been imported/retrieved into the device within Expedition. Some stuff I've done/is configured:
I've also got another thread out there regarding the "process Enabled Files" option that is greyed out in this context. The only way I can process these logs is by letting the daily processing schedule catch up to them, or manually changing that schedule to be 2 min from now, for instance.
In any case, the server quickly fills up with space as logs aren't being deleted after processing. My thinking is that logs are uploaded at 1600, ACL changed at 0000, then auto processing kicking off at 0400. So far it seems to all work except the deleting part. Any tips?
09-28-2020 08:47 AM
Is this a Panorama thing? The permissions on these files look correct, but the auto processing phase never deletes them.
09-28-2020 08:50 AM
@BenKnorr2 Have you tried the below commands already?
sudo chown expedition.www-data/home/expedition/logs
sudo chmod -r 770/home/expedition/logs
09-28-2020 09:41 AM
Have you verified that the device that reported those logs has the After Process action set to delete?
Maybe you had it for the Panorama, but the device says something else, and we will take action based on what the device states.
09-29-2020 11:32 AM
Thanks for the response. Unfortunately, no, permissions have been consistent and what appears to be correct since beginning. In Expedition GUI / Settings /M.Learning, there is a blurb at the bottom for CSV and file rights:
Scheduled Log Export from FW devices may export your log files as expedition owned files.
In case you want to activate delete after processing the CSV logs, make sure that www-data has write rights over the files.
You can achieve it by adding the following into your root cron (the following example will verify the file rights every day at 00:05 am):
00 05 * * * * php /var/www/html/OS/spark/scripts/changeCSVLogRights.php
I entered this in my crontab (syntax shown above looks incorrect though; has an extra *) and it fixes permissions for 660 / chown www-data:expedition for any logs in a project that are set to be autoprocessed but haven't been processed yet. This does work. This step helped in my lab environment to get autoprocessing+deleteafter to succeed.
For what it's worth, here are two log files: one with 770 was set immediately after it was uploaded to expedition prior to processing. The other was set using the built-in script that runs at 00:05 nightly via cron. Both files have since been "auto processed" with "delete after processing" checked, but no deletion occurred after processing. Just showing that 770/660 seem to have no impact on problem.
expedition@Expedition:~/logs$ ls -lah
drwxrwxr-x 2 expedition expedition 4.0K Sep 28 15:56 .
drwxr-xr-x 5 expedition expedition 4.0K Sep 10 08:41 ..
-rwxrwx--- 1 expedition www-data 157G Sep 27 17:39 PA5250-PRI_traffic_2020_09_27_last_calendar_day.csv
-rw-rw---- 1 expedition www-data 161G Sep 28 17:50 PA5250-PRI_traffic_2020_09_28_last_calendar_day.csv
My lab environment has a FW sending logs via SCP to expedition, and the device in expedition is for the firewall itself w/no panorama. "process enabled files" button in device tab works for manual processing (have another thread on this board about this part). Logs weren't getting deleted until I put the php script in crontab, now it works good. My prod environment has FW sending logs via SCP to expedition, but the device is panorama with managed devices/config imported to it. "process enabled files" is greyed out and has never been usable, for what its worth, and deleting logs after auto processing has never worked.
09-29-2020 11:47 AM
For Panorama managed FW, you will need to check the FW ML setting, go to the "Device" tab, click on the "show all devices" icon as on the right upper corner as shown in the below screenshot, and find the FW that's matching your traffic logs, check the ML setting on that firewall.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!