Hello. I'm looking for suggestions for what to log out to ELK. I understand that this will be different for every environment and should be based on existing log retention policies etc. However, we have no such policies, and I am currently logging far too much, I'm sure. In order to get traffic visibility > 2 days, I rolled up an ELK stack on a server with 17TB storage, and pointed all system, traffic, threat, and config logs to it. I setup every security policy to forward to this collector, and the traffic logs, in just a few days, have grown to over 129GB. The space is there to support that. However, Elasticsearch isn't happy and extracting the data is extremely slow. It could be that I need to tune the collector, but I think more likely, I need to be more realistic/logical about what I'm logging out. I would appreciate any general suggestions. For example, is logging blocked traffic useful enough to justify it? I can see where it may be useful in an audit trail, but if it was blocked/dropped, how much should we care about it?
... View more