MineMeld - CSV input feature

Reply
L1 Bithead

MineMeld - CSV input feature

Hi all,

 

Firstly, great work on MineMeld - it is fantastic!!! I have it working great for dynamic IP lists and AF export lists, but our customer would like to import Indicators from CSV. It doesn't look possible with current class/prototypes. Any suggestions? I could script the import to a hosted list, but kind of defeats the objective.

 

Additionally, is there a way to modify the condifence value of AF indicators (from the 75 default). Assume it's best to just manipulate on the output node?

 

Thanks,

Tim

Tags (2)
L7 Applicator

Re: MineMeld - CSV input feature

Hi Tim,

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. I have opened an ER to support CSV upload: minemeld-core#30.

 

To change the confidence level of AF indicators you have to create a new prototype. That's easy:

 

1. Go to CONFIG > BROWSE PROTOTYPES and select the autofocus.exportList prototype

2. Click on NEW

1469015931239screensave.png

 

3. Inside the prototype editor change the confidence level as in picture. In this case the confidence level has been increased to 80

 

1469016013193screensave.png

 

4. Save the prototype bt pressing OK

5. Use the new prototype to create a new node inside the config.

L1 Bithead

Re: MineMeld - CSV input feature

Thank Luigi that's great


@lmori wrote:

Hi Tim,

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. I have opened an ER to support CSV upload: minemeld-core#30.

 

To change the confidence level of AF indicators you have to create a new prototype. That's easy:

 

1. Go to CONFIG > BROWSE PROTOTYPES and select the autofocus.exportList prototype

2. Click on NEW

1469015931239screensave.png

 

3. Inside the prototype editor change the confidence level as in picture. In this case the confidence level has been increased to 80

 

1469016013193screensave.png

 

4. Save the prototype bt pressing OK

5. Use the new prototype to create a new node inside the config.


 

L2 Linker

Re: MineMeld - CSV input feature


@lmori wrote:

 

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. 

 

 


 

Hi! I'm wondering if you can elaborate on how I would setup a node to accept input from a file uploaded via SFTP (or otherwise placed on the hard drive). I ask because I'd like to build a node that can import data from a proprietary feed we subscribe to, and the only way to get data from that feed is via a CLI utility that can spit out either a pipe or CSV delimited ouput. There's no URL that can be polled directly, unfortunately. 

 

I figured I could run the CLI utility on the minemeld server and have it output in the correct format to a location where it would be picked up for processing. Is that do-able?

Thanks!

Dan

Highlighted
L7 Applicator

Re: MineMeld - CSV input feature

Hi Dan,

currently the CSV miner accepts only HTTP/HTTPS URLs as source, but we could extend it to load local files. Would you run the CLI utility directly on the MIneMeld VM ?

 

Luigi

L3 Networker

Re: MineMeld - CSV input feature

I second and third this request! It would be great to have a CLI tool until we could get a nice webpage for this. Eventually adding an upload tool to the nodes page under the indicator tab.

 

In the meantime, could we get a quick step-by-step on how to put together the YAML format for manually adding indicators in bulk?

 

Thanks!

L3 Networker

Re: MineMeld - CSV input feature

Can we just go in and edit the .YML files under /opt/minemeld/local/config/? I see an example here:

 

nano  /opt/minemeld/local/config/my_manual_miner_indicators.yml


- {indicator: 1.2.3.4, share_level: green, type: IPv4}
- {indicator: 5.6.7.8, share_level: green, type: IPv4}
- {comment: 'InfoSec request, Work Order #12345', indicator: 1.2.3.4, share_level: green}
- {comment: 'InfoSec request, Work Order #12345', indicator: 5.6.7.8, share_level: green}

 

Do we need to restart anything when we're done adding our changes?

 

-Nasir

L3 Networker

Re: MineMeld - CSV input feature

That was a bad idea...I must have missed a step:

2016-08-02T12:19:57 (13568)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2016-08-02T12:19:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:20:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:21:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:22:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:23:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:24:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:25:58 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:26:58 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'

 

After updating the YML file with indicators and comments, do I need to add anything additional to the file or update a reference elsewhere?

L7 Applicator

Re: MineMeld - CSV input feature

Hi Nasir,

format looks good and I have tested it and it works for me. You sure you haven't made mistakes in saving the file ?

Once saved you can force the Miner to reload the indicators using:

/opt/minemeld/engine/current/bin/mm-console my_manual_miner hup
L3 Networker

Re: MineMeld - CSV input feature

Luigi,

 

I found my issue. I did a bulk find and replace on a flat list of IP's to build the YML, and inadvertently added some space. I took a list of IP's like this:

 

1.1.1.1

2.2.2.2

3.3.3.3

 

and then in notepad++ used this regex to build the YML which was added to my existing my_manual_miner_indicators.yml:

 

Find:

(.*)\r\n

 

Replace: 

- {indicator: \1, share_level: green, type: IPv4}\r\n- {comment: 'See ServiceNow Incident #123 - 20160801', indicator: \1, share_level: green}\r\n

 

However, I had accidentally added two spaces between "indicator: \1," and "share_level: green", which apparently broke the YML parser! I fixed it and it's now working for me as well. Thanks as always for your prompt responses and awesome tool!

 

NOTE: The find and replace statements above are correct, so readers can feel free to use them. I removed the offending extra whitespace that caused the issue referenced above.

 

Regards,

Nasir

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!