Help on CSV Output Feed

[kx1499]

I'm looking to output feeds to a format that I can ingest in some log analysis tools, and need to output fields that I have defined in miners. Is there any information on how to access that data and output it?

[lmori]

Could give us more details about the format and fields your log analsys platform could ingest ?

[kx1499]

Sure. Thanks for the quick response.

So we have some custom scripts in place that parse any number of feeds, normalize the data, and feed it into our log analysis solution. For each feed we extract date, source_name, ioc_desc, ioc. We then add category, ioc_type, confidence. I'm looking to replace soem of our custom scripts with this, but I'm having trouble getting the data back out in that format. 

I can do teh parsing just fine with teh miners, filter/aggregate with the processors, but I can't get the output to limit the fields and send it in tsv/csv/syslog format as is. Poking around in the flask part of the code I was able to tweak the feedredis.py to output in tsv  - just not sure if thats the best way to do it. Also a little stuck on how to specify field filters in teh output. 

[lmori]

Hi kx1499,

feedredis.py could be the best place to do this. I think the best way to restrict field is sending the required fields as a URL parameter. Do you have an example of the tsv output format required by your log analysis tool ? Is the log analysis tool a commercial solution ?

Thanks,

luigi