MineMeld - CSV input feature

[tkirk]

Hi all,

Firstly, great work on MineMeld - it is fantastic!!! I have it working great for dynamic IP lists and AF export lists, but our customer would like to import Indicators from CSV. It doesn't look possible with current class/prototypes. Any suggestions? I could script the import to a hosted list, but kind of defeats the objective.

Additionally, is there a way to modify the condifence value of AF indicators (from the 75 default). Assume it's best to just manipulate on the output node?

Thanks,

Tim

[lmori]

Hi Tim,

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. I have opened an ER to support CSV upload: minemeld-core#30.

To change the confidence level of AF indicators you have to create a new prototype. That's easy:

1. Go to CONFIG > BROWSE PROTOTYPES and select the autofocus.exportList prototype

2. Click on NEW

3. Inside the prototype editor change the confidence level as in picture. In this case the confidence level has been increased to 80

4. Save the prototype bt pressing OK

5. Use the new prototype to create a new node inside the config.

[tkirk]

Thank Luigi that's great

---

@lmori wrote:

Hi Tim,

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. I have opened an ER to support CSV upload: minemeld-core#30.

To change the confidence level of AF indicators you have to create a new prototype. That's easy:

1. Go to CONFIG > BROWSE PROTOTYPES and select the autofocus.exportList prototype

2. Click on NEW

3. Inside the prototype editor change the confidence level as in picture. In this case the confidence level has been increased to 80

4. Save the prototype bt pressing OK

5. Use the new prototype to create a new node inside the config.

---

[DanWoodruff]

---

@lmori wrote:

currently (0.9.18) the only way to import indicators via CSV is translate the file into YAML and then upload it via SFTP or via MineMeld web API. 

---

Hi! I'm wondering if you can elaborate on how I would setup a node to accept input from a file uploaded via SFTP (or otherwise placed on the hard drive). I ask because I'd like to build a node that can import data from a proprietary feed we subscribe to, and the only way to get data from that feed is via a CLI utility that can spit out either a pipe or CSV delimited ouput. There's no URL that can be polled directly, unfortunately. 

I figured I could run the CLI utility on the minemeld server and have it output in the correct format to a location where it would be picked up for processing. Is that do-able?

Thanks!

Dan

[lmori]

Hi Dan,

currently the CSV miner accepts only HTTP/HTTPS URLs as source, but we could extend it to load local files. Would you run the CLI utility directly on the MIneMeld VM ?

Luigi

[nbilal]

I second and third this request! It would be great to have a CLI tool until we could get a nice webpage for this. Eventually adding an upload tool to the nodes page under the indicator tab.

In the meantime, could we get a quick step-by-step on how to put together the YAML format for manually adding indicators in bulk?

Thanks!

[nbilal]

Can we just go in and edit the .YML files under /opt/minemeld/local/config/? I see an example here:

nano  /opt/minemeld/local/config/my_manual_miner_indicators.yml

- {indicator: 1.2.3.4, share_level: green, type: IPv4}
- {indicator: 5.6.7.8, share_level: green, type: IPv4}
- {comment: 'InfoSec request, Work Order #12345', indicator: 1.2.3.4, share_level: green}
- {comment: 'InfoSec request, Work Order #12345', indicator: 5.6.7.8, share_level: green}

Do we need to restart anything when we're done adding our changes?

-Nasir

[nbilal]

That was a bad idea...I must have missed a step:

2016-08-02T12:19:57 (13568)base.state INFO: wlWhiteListIPv4 - transitioning to state 4
2016-08-02T12:19:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:20:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:21:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:22:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:23:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:24:57 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:25:58 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'
2016-08-02T12:26:58 (13562)amqp._in_callback ERROR: Error in RPC reply from mbus:slave:my_manual_miner: 'YamlIPv4FT' object has no attribute 'table'

After updating the YML file with indicators and comments, do I need to add anything additional to the file or update a reference elsewhere?

[lmori]

Hi Nasir,

format looks good and I have tested it and it works for me. You sure you haven't made mistakes in saving the file ?

Once saved you can force the Miner to reload the indicators using:

/opt/minemeld/engine/current/bin/mm-console my_manual_miner hup

[nbilal]

Luigi,

I found my issue. I did a bulk find and replace on a flat list of IP's to build the YML, and inadvertently added some space. I took a list of IP's like this:

1.1.1.1

2.2.2.2

3.3.3.3

and then in notepad++ used this regex to build the YML which was added to my existing my_manual_miner_indicators.yml:

Find:

(.*)\r\n

Replace: 

- {indicator: \1, share_level: green, type: IPv4}\r\n- {comment: 'See ServiceNow Incident #123 - 20160801', indicator: \1, share_level: green}\r\n

However, I had accidentally added two spaces between "indicator: \1," and "share_level: green", which apparently broke the YML parser! I fixed it and it's now working for me as well. Thanks as always for your prompt responses and awesome tool!

NOTE: The find and replace statements above are correct, so readers can feel free to use them. I removed the offending extra whitespace that caused the issue referenced above.

Regards,

Nasir

[lmori]

This way of expressing the list in YAML is a bit more user friendly and it works as well:

- indicator: "1.2.3.4"
  share_level: green
  type: IPv4
- indicator: "5.6.7.8"
  share_level: green
  type: IPv4
- comment: "InfoSec request, Work Order #12345"
  indicator: "1.2.3.4"
  share_level: green
- comment: "InfoSec request, Work Order #12345"
  indicator: "5.6.7.8"
  share_level: green

[maltwist]

Hi lmori,

  Can you explain how to acces "MineMeld web API" 

thanks

Maltwist!

[DanWoodruff]

---

@lmori wrote:

Hi Dan,

currently the CSV miner accepts only HTTP/HTTPS URLs as source, but we could extend it to load local files. Would you run the CLI utility directly on the MIneMeld VM ?

Luigi

---

Hi Luigi,

Yep, most likely. That'll keep all my stuff for this initiative on the same machine.

However, I think what Nasir described will work just fine for us right now. I'm going to give that a try and see how I make out. I can always adapt to a cleaner method as MineMeld evolves in the future.

Thanks for the great tool!

[lmori]

Thanks Dan, let me know if something does not work. Happy to help.

Luigi

[nbilal]

thanks for the useful command! I have implemented it into my script. However, it looks like the correct syntax is:

/opt/minemeld/engine/current/bin/mm-console hup my_manual_miner