Help on CSV Output Feed

L0 Member

Help on CSV Output Feed

I'm looking to output feeds to a format that I can ingest in some log analysis tools, and need to output fields that I have defined in miners. Is there any information on how to access that data and output it?

L7 Applicator

Re: Help on CSV Output Feed

Could give us more details about the format and fields your log analsys platform could ingest ?

L0 Member

Re: Help on CSV Output Feed

Sure. Thanks for the quick response.

 

So we have some custom scripts in place that parse any number of feeds, normalize the data, and feed it into our log analysis solution. For each feed we extract date, source_name, ioc_desc, ioc. We then add category, ioc_type, confidence. I'm looking to replace soem of our custom scripts with this, but I'm having trouble getting the data back out in that format. 

 

I can do teh parsing just fine with teh miners, filter/aggregate with the processors, but I can't get the output to limit the fields and send it in tsv/csv/syslog format as is. Poking around in the flask part of the code I was able to tweak the feedredis.py to output in tsv  - just not sure if thats the best way to do it. Also a little stuck on how to specify field filters in teh output. 

 

 

L7 Applicator

Re: Help on CSV Output Feed

Hi kx1499,

feedredis.py could be the best place to do this. I think the best way to restrict field is sending the required fields as a URL parameter. Do you have an example of the tsv output format required by your log analysis tool ? Is the log analysis tool a commercial solution ?

 

Thanks,

luigi

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!