10-20-2011 04:40 AM
Hi - I know there are plenty of discussion about log size - but I don't think anything answers my query.
We installed out Palo Altos (2x 4050s) at the weekend. They sit between our internal datacentres and the rest of the customers network - so they are not internet facing. We log traffic (at start and end) and threat logs (alert on AV/Spy/Vulnerability - no others are enabled) and obviously config and system logs.
We are in active/passive mode (so traffic logs only come from the active PA4050). We forward all logs to Panorama - which is installed in ESX with 1TB of space (we estimated this would give us about 9 months worth of logs). We would then export via the scheduled log export direct from the firewalls a log each day to ensure we kept the logs in some format for the 13months we're required to keep them.
From a logdb-quota on the Panorama we're currently running at around 9.5GB per day of traffic logs alone totalling 34GB since Sunday (threat is only around 70MB - so tiny in comparison).
My thoughts and questions are:
1. Looking at the log quotas on Panorama and the useage of other logs so far - it looks like I could quite happily up the quota for traffic from the 25% (232GB) allocated to 75% allocation - and still leave the other logs OK. Would anyone advise against this?
2. Even if I do that that I would estimate only 74 days or so of logs on Panorama (only two 1/2 months worth - rather than the 9 months we had estimated).,
3. Even if I could keep 9 months on Panorama - the plan was to export CSV from the Palo Alto direct to an FTP server via the scheduled log export so we have them for the 13 months required. Regarding the CSV export - I've increased the max size of the csv file to the max allowed lines of 1048576 - the currently stored logs on the PA4050 itself is 32GB - if I export a this - it will never fit in a single csv file - does it split it into several files? - or does it just bin anything larger than the number of lines? Even if I export only a days worth at 9.5GB - this would also be too large for a single CSV file - how does the PA4050 handle days worth of logs of this size?
4. I've tried a manual test of creating of a CSV file from the traffic logs screen of my passive PA4050 (which has only about 76MB worth in it's database) - the export started - but the link it takes me to on the PA4050 failed:
Firefox can't find the file at https://10.245.57.125/php/monitor/log.export.csv.php?filename=/opt/pancfg/session/pan/csv/7320085078.... Is this due to size or a bug do you think?
Apologies for the long and many tentacled nature of the questions!
Any help most appreciated!
10-20-2011 07:48 AM
Trying to use CSV log export to backup logs daily is not scalable at high logging rates. I do not think you will be able to get scheduled log export working in a way which accomplishes your goal at the logging rates you mentioned. One option could be to selectively forward the logs you must retain only and let other logs age out on devices. This will depend on your requirements which sometimes included considerations with respect to compliance, regulations, and auditing.
I would recommend using a Syslog collector solution and/or Panorama to meet your long term retention requirements.
Here is an article on how to estimate the amount of space you need in Panorama to meet your goals.
10-21-2011 03:59 AM
Many thanks mschuricht. That mirrored the conclusion I'd come to. We can't turn off logging of any rules as the customer requires that full view. We plan to syslog now to capture that with log rotation every hour.
NB - I've run the sizing calculation from the link - I estimated around 80TB for our needs!! Do you see any issues with latering the quotas in give us more space available for the traffic logs so that we at least keep the logs for longer on the Panorama? I can't see a problem myself.
10-21-2011 08:39 AM
There should be no issues adjusting the quota as long as the aggregate total of all log types is <= 100%.
10-25-2011 03:38 PM
Hi - Following on from this. When I try to adjust the values in the log-storage values for Panorama in the setup screen it won't allow me to do so. It keeps coming back with an error of "management is missing 'storage-partition' ". I initially thought it was because I was altering it drastically. However even after keeping the change simple - taking 5% off the threat log and adding 5% to the traffic log even this fails with the above error. I've not tried from the CLI yet - but is there any explanation for this?
10-27-2011 03:58 AM
Bump - anyone help on this. At the moment we can't alter the logdb settings (at least on the GUI) - we get an error of "management is missing 'storage-partition' " even if we make a tiny change eg. reducing the size of the threat databse to 24% from 25%.
10-31-2011 09:34 PM
Try the following and let us know if this doesn't resolve your issue.
Go to Panorama Tab -> Storage Partition setup -> Change to NFS Hit OK -> Go back into Storage Partition Setup and switch back to Internal -> Hit OK. Try the commit again.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
