topic Re: Help on CSV Output Feed in General Topics

Help on CSV Output Feed

kx1499 — Thu, 28 Jul 2016 02:23:26 GMT

I'm looking to output feeds to a format that I can ingest in some log analysis tools, and need to output fields that I have defined in miners. Is there any information on how to access that data and output it?

Re: Help on CSV Output Feed

lmori — Thu, 28 Jul 2016 08:23:03 GMT

Could give us more details about the format and fields your log analsys platform could ingest ?

Re: Help on CSV Output Feed

kx1499 — Fri, 29 Jul 2016 01:02:48 GMT

Sure. Thanks for the quick response.

So we have some custom scripts in place that parse any number of feeds, normalize the data, and feed it into our log analysis solution. For each feed we extract date, source_name, ioc_desc, ioc. We then add category, ioc_type, confidence. I'm looking to replace soem of our custom scripts with this, but I'm having trouble getting the data back out in that format.

I can do teh parsing just fine with teh miners, filter/aggregate with the processors, but I can't get the output to limit the fields and send it in tsv/csv/syslog format as is. Poking around in the flask part of the code I was able to tweak the feedredis.py to output in tsv - just not sure if thats the best way to do it. Also a little stuck on how to specify field filters in teh output.

Re: Help on CSV Output Feed

lmori — Fri, 29 Jul 2016 07:55:48 GMT

Hi kx1499,

feedredis.py could be the best place to do this. I think the best way to restrict field is sending the required fields as a URL parameter. Do you have an example of the tsv output format required by your log analysis tool ? Is the log analysis tool a commercial solution ?

Thanks,

luigi