Log Retention Policies and Best Practices

[hburton]

Hello.

I'm looking for suggestions for what to log out to ELK. I understand that this will be different for every environment and should be based on existing log retention policies etc. However, we have no such policies, and I am currently logging far too much, I'm sure.

In order to get traffic visibility > 2 days, I rolled up an ELK stack on a server with 17TB storage, and pointed all system, traffic, threat, and config logs to it. I setup every security policy to forward to this collector, and the traffic logs, in just a few days, have grown to over 129GB. The space is there to support that. However, Elasticsearch isn't happy and extracting the data is extremely slow. It could be that I need to tune the collector, but I think more likely, I need to be more realistic/logical about what I'm logging out.

I would appreciate any general suggestions. For example, is logging blocked traffic useful enough to justify it? I can see where it may be useful in an audit trail, but if it was blocked/dropped, how much should we care about it?

[PBowden]

Hi,

Thanks for reaching out. Generally, compliance regulations would be a good starting place ie SOX, PCI, FISMA, HIPAA, FERPA, ISO 27001. 

[hburton]

Thanks Patrick.

I appreciate your response.

I haven't read through all of these guidelines, because I've only been reading for about 35 years, but I have googled both PCI and HIPAA log retention policies and they are about as vague as standards/guidelines tend to be. They tend to state how long to maintain logs, and how to store them pretty concisely. However, for which logs to store, they say "user logins, ..., firewalls logs." 

What I'm hoping to get is an idea which logs people tend to care about, so that I can whittle down what I'm sending to log management. My log stream is around 5Mb/s right now.

The Palo Alto firewall isn't the only system logging to it, but it's one of just a few, and the others log a handful of entries per minute, so that 5Mb/s is almost entirely my firewall logs.

This is also the reason that we aren't logging these to Splunk. The firewalls alone would chew up our total ingress budget.

[PBowden]

I see. In my experience, I would be looking for authorization logs from domain controllers, and Linux authorization logs if the environment warranted. Also, application logs are great to have. Additionally, consider applications sometimes have alternative logs; so you would want to know where those go and what they look like. Then, after my topics were defined I would proceed categorizing what events are relevant to me and my environment. Finally, use those events to determine threat. Furthermore, you can also use said events to filter down your overall logs in ways such as, logins and/or failed logins. Is this information of more assistance?

[hburton]

Both answers have been helpful. Thank you.

I'm mostly interested in the Palo Alto logs though. Specifically, the traffic logs. The traffic logs are 99+% of what I'm currently sending to log management, and the bulk of the load that I would like to reduce. Currently, every security policy is configured to log and forward those logs to log management. I feel like that is probably more than what we realistically need, and I'm curious what other people are keeping.

[PBowden]

Hi @hburton ,

Im working on getting a better answer for you. Standby.