Forward logs from Firewall to Expedition for ML but storage always full

[EdwinKhng]

Hi All,

I tried to forward traffic logs from firewalls to the expedition for machine learning purposes, however, those traffic logs always occupied the storage and cause my expedition always unable to login with the WebUI. After I free up some storage and restart the MariaDB then I will only able to login.

However, when I tried to retrieve the Machine Learning results, it show no logs and no results. Ideally, I got one Allow-Any rules inside the firewall and I want to analyze those traffic which hitted the policy through machine learning and generate the results in csv. Previously I able to do once successfully when the firewalls and expedition just setup, after 6 month later I found even one logs will occupied the whole storage on expedition VM.

My question is :

1. Can I just forward the last 7 days logs of those traffic that hitted the Allow-Any rules to expedition for Machine Learning purpose only?

2. How can I get the Machine Learning analyze results without the logs on expedition? Can I directly pull traffic logs on firewalls?

Thank you very much for the helps!

[lychiang]

Hi @EdwinKhng ,

To answer your questions:

1. Yes, you could, there are two ways to do it.  1. you can specify query like (rule eq Allow-Any) in the traffic log window and download the log manually from the firewall 2. you could make Expedition as Syslog server and use log forwarding profile in the "Allow-Any"'' rule. 

2. For ML, You will need the logs stored in Expedition, however, if you use Rule Enrichment, you can add firewall as log connector , then you don't need to store logs in Expedition.