topic Re: MineMeld - CSV input feature in General Topics

MineMeld - CSV input feature

tkirk — Tue, 19 Jul 2016 15:12:20 GMT

Hi all,

Firstly, great work on MineMeld - it is fantastic!!! I have it working great for dynamic IP lists and AF export lists, but our customer would like to import Indicators from CSV. It doesn't look possible with current class/prototypes. Any suggestions? I could script the import to a hosted list, but kind of defeats the objective.

Additionally, is there a way to modify the condifence value of AF indicators (from the 75 default). Assume it's best to just manipulate on the output node?

Thanks,

Tim

Re: MineMeld - CSV input feature

lmori — Wed, 20 Jul 2016 12:06:01 GMT

Hi Tim,

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. I have opened an ER to support CSV upload: minemeld-core#30.

To change the confidence level of AF indicators you have to create a new prototype. That's easy:

1. Go to CONFIG > BROWSE PROTOTYPES and select the autofocus.exportList prototype

2. Click on NEW

3. Inside the prototype editor change the confidence level as in picture. In this case the confidence level has been increased to 80

4. Save the prototype bt pressing OK

5. Use the new prototype to create a new node inside the config.

Re: MineMeld - CSV input feature

tkirk — Thu, 21 Jul 2016 18:39:56 GMT

Thank Luigi that's great

@lmori wrote:

Hi Tim,

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. I have opened an ER to support CSV upload: minemeld-core#30.

To change the confidence level of AF indicators you have to create a new prototype. That's easy:

1. Go to CONFIG > BROWSE PROTOTYPES and select the autofocus.exportList prototype

2. Click on NEW

3. Inside the prototype editor change the confidence level as in picture. In this case the confidence level has been increased to 80

4. Save the prototype bt pressing OK

5. Use the new prototype to create a new node inside the config.

Re: MineMeld - CSV input feature

DanWoodruff — Fri, 29 Jul 2016 20:30:13 GMT

@lmori wrote:

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API.

Hi! I'm wondering if you can elaborate on how I would setup a node to accept input from a file uploaded via SFTP (or otherwise placed on the hard drive). I ask because I'd like to build a node that can import data from a proprietary feed we subscribe to, and the only way to get data from that feed is via a CLI utility that can spit out either a pipe or CSV delimited ouput. There's no URL that can be polled directly, unfortunately.

I figured I could run the CLI utility on the minemeld server and have it output in the correct format to a location where it would be picked up for processing. Is that do-able?

Thanks!

Dan

Re: MineMeld - CSV input feature

lmori — Mon, 01 Aug 2016 17:46:25 GMT

Hi Dan,

currently the CSV miner accepts only HTTP/HTTPS URLs as source, but we could extend it to load local files. Would you run the CLI utility directly on the MIneMeld VM ?

Luigi

Re: MineMeld - CSV input feature

nbilal — Tue, 02 Aug 2016 11:23:38 GMT

I second and third this request! It would be great to have a CLI tool until we could get a nice webpage for this. Eventually adding an upload tool to the nodes page under the indicator tab.

In the meantime, could we get a quick step-by-step on how to put together the YAML format for manually adding indicators in bulk?

Thanks!

Re: MineMeld - CSV input feature

nbilal — Tue, 02 Aug 2016 12:07:21 GMT

Can we just go in and edit the .YML files under /opt/minemeld/local/config/? I see an example here:

nano /opt/minemeld/local/config/my_manual_miner_indicators.yml

- {indicator: 1.2.3.4, share_level: green, type: IPv4}
- {indicator: 5.6.7.8, share_level: green, type: IPv4}
- {comment: 'InfoSec request, Work Order #12345', indicator: 1.2.3.4, share_level: green}
- {comment: 'InfoSec request, Work Order #12345', indicator: 5.6.7.8, share_level: green}

Do we need to restart anything when we're done adding our changes?

-Nasir

Re: MineMeld - CSV input feature

nbilal — Tue, 02 Aug 2016 12:29:50 GMT

That was a bad idea...I must have missed a step:

2016-08-02T12:19:57 (13568)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2016-08-02T12:19:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:20:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:21:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:22:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:23:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:24:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:25:58 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:26:58 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'

After updating the YML file with indicators and comments, do I need to add anything additional to the file or update a reference elsewhere?

Re: MineMeld - CSV input feature

lmori — Tue, 02 Aug 2016 13:10:19 GMT

Hi Nasir,

format looks good and I have tested it and it works for me. You sure you haven't made mistakes in saving the file ?

Once saved you can force the Miner to reload the indicators using:

/opt/minemeld/engine/current/bin/mm-console my_manual_miner hup

Re: MineMeld - CSV input feature

nbilal — Tue, 02 Aug 2016 13:15:57 GMT

Luigi,

I found my issue. I did a bulk find and replace on a flat list of IP's to build the YML, and inadvertently added some space. I took a list of IP's like this:

1.1.1.1

2.2.2.2

3.3.3.3

and then in notepad++ used this regex to build the YML which was added to my existing my_manual_miner_indicators.yml:

Find:

(.*)\r\n

Replace:

- {indicator: \1, share_level: green, type: IPv4}\r\n- {comment: 'See ServiceNow Incident #123 - 20160801', indicator: \1, share_level: green}\r\n

However, I had accidentally added two spaces between "indicator: \1," and "share_level: green", which apparently broke the YML parser! I fixed it and it's now working for me as well. Thanks as always for your prompt responses and awesome tool!

NOTE: The find and replace statements above are correct, so readers can feel free to use them. I removed the offending extra whitespace that caused the issue referenced above.

Regards,

Nasir

Re: MineMeld - CSV input feature

lmori — Tue, 02 Aug 2016 15:18:48 GMT

This way of expressing the list in YAML is a bit more user friendly and it works as well:

- indicator: "1.2.3.4"
  share_level: green
  type: IPv4
- indicator: "5.6.7.8"
  share_level: green
  type: IPv4
- comment: "InfoSec request, Work Order #12345"
  indicator: "1.2.3.4"
  share_level: green
- comment: "InfoSec request, Work Order #12345"
  indicator: "5.6.7.8"
  share_level: green

Re: MineMeld - CSV input feature

maltwist — Thu, 04 Aug 2016 14:38:28 GMT

Hi lmori,

Can you explain how to acces "MineMeld web API"

thanks

Maltwist!

Re: MineMeld - CSV input feature

DanWoodruff — Tue, 09 Aug 2016 19:40:48 GMT

@lmori wrote:

Hi Dan,

currently the CSV miner accepts only HTTP/HTTPS URLs as source, but we could extend it to load local files. Would you run the CLI utility directly on the MIneMeld VM ?

Luigi

Hi Luigi,

Yep, most likely. That'll keep all my stuff for this initiative on the same machine.

However, I think what Nasir described will work just fine for us right now. I'm going to give that a try and see how I make out. I can always adapt to a cleaner method as MineMeld evolves in the future.

Thanks for the great tool!

Re: MineMeld - CSV input feature

lmori — Wed, 10 Aug 2016 12:48:19 GMT

Thanks Dan, let me know if something does not work. Happy to help.

Luigi

Re: MineMeld - CSV input feature

nbilal — Tue, 08 Nov 2016 14:59:24 GMT

thanks for the useful command! I have implemented it into my script. However, it looks like the correct syntax is:

/opt/minemeld/engine/current/bin/mm-console hup my_manual_miner