Log Retention Policies and Best Practices

Showing results for 
Show  only  | Search instead for 
Did you mean: 

Log Retention Policies and Best Practices

L1 Bithead


I'm looking for suggestions for what to log out to ELK. I understand that this will be different for every environment and should be based on existing log retention policies etc. However, we have no such policies, and I am currently logging far too much, I'm sure.


In order to get traffic visibility > 2 days, I rolled up an ELK stack on a server with 17TB storage, and pointed all system, traffic, threat, and config logs to it. I setup every security policy to forward to this collector, and the traffic logs, in just a few days, have grown to over 129GB. The space is there to support that. However, Elasticsearch isn't happy and extracting the data is extremely slow. It could be that I need to tune the collector, but I think more likely, I need to be more realistic/logical about what I'm logging out.


I would appreciate any general suggestions. For example, is logging blocked traffic useful enough to justify it? I can see where it may be useful in an audit trail, but if it was blocked/dropped, how much should we care about it?


Hi @hburton ,


Im working on getting a better answer for you. Standby. 


Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!