how to write a simple miner documentation

[vb0398]

Hi there,

   I'm a new user, so hopefully this is a simple question.

I installed minemeld via source code on ubuntu 14.04 using the instructions on this page : 

https://github.com/PaloAltoNetworks/minemeld-ansible

 The installation went smoothly and there were no errors.

I then went through the exercise of writing a test miner using these instructions :  https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner

I create the ytexample.py file in the detailed directory, replaced /opt/minemeld/local/config/committed-config.yml with the node information available in the "How-To.." webpage, and restarted the minemeld service.  From this point, I check the minemeld-engine.log file, and I see the following error:

minemeld-engine.log:2017-05-15T23:46:45 (14879)config._load_and_validate_config_from_file ERROR: Invalid config /opt/minemeld/local/config/committed-config.yml: Unknown node class minemeld.ft.ytexample.YTExample in testYT

Has anybody seen this error before?

Thanks...

[lmori]

Hi @vb0398,

that guide should be updated, there are 2 additional steps:

- open the file nodes.json in the main directory of the minemeld engine and add the following to the dictionary of nodes:

[...]
},
"minemeld.ft.ytexample.YTExample": {"class": "minemeld.ft.ytexample.YTExample"}
}

- run "/opt/minemeld/engine/current/bin/pip install -e /opt/minemeld/engine/core"

NOTE: If you are looking into creating a new Miner my suggestion is to use external extensions, they are more flexible and agile. The same code of the Miner in the guide has been packaged as an extension here:

https://github.com/PaloAltoNetworks/youtube-miner

[vb0398]

Hi @lmori - Thanks for your reply.

I did those last 2 steps, and it still doesn't work.  The error this time in the minemeld-engine.log :

ImportError: No module named YTExample
ImportError: No module named YTExample

I will go ahead and try the extension route.

Best,

[lmori]

Hi @vb0398,

it seems that python is not able to find the YTExample module containing the Miner class.

Please could you attach the nodes.json file ? is the YTExample.py in minemeld/ft directory ?

Thanks,

luigi

[vb0398]

hi @lmori,

   In your documentation, it says to name the file, 'ytexample.py' - all lowercase, and that is the name of the file in the "/opt/minemeld/engine/core/minemeld/ft" directory.  

Attached the nodes.json file

[lmori]

Hi @vb0398,

sorry for the late reply. There is a typo in you nodes.json file, the line should read:

[...]
    "minemeld.ft.ytexample.YTExample": {
        "class": "minemeld.ft.ytexample:YTExample"
    }
[...]

instead in your file you have:

[...]
    "minemeld.ft.ytexample.YTExample": {
        "class": "minemeld.ft.ytexample.YTExample"
    }
[...]

(":" is a Python thing)

[vb0398]

Ah - ok - corrected nodes.json - same error...

  "class": "minemeld.ft.threatq:Export"
},
"minemeld.ft.tmt.DTIAPI": {
"class": "minemeld.ft.tmt:DTIAPI"
},
"minemeld.ft.vt.Notifications": {
"class": "minemeld.ft.vt:Notifications"
},
"minemeld.ft.mm.JSONSEQMiner": {
"class": "minemeld.ft.mm:JSONSEQMiner"
},
"minemeld.ft.ytexample.YTExample": {
"class": "minemeld.ft.ytexample.YTExample"
}
}

error:

...

2017-05-28T23:57:10 (6730)launcher.main INFO: multiprocessing: #cores: 1
2017-05-28T23:57:10 (6730)launcher.main INFO: multiprocessing: max #chassis: 1
2017-05-28T23:57:10 (6730)launcher.main INFO: Number of chassis: 1
2017-05-28T23:57:10 (6734)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ytexample.YTExample
2017-05-28T23:57:10 (6734)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/core/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/core/minemeld/ft/__init__.py", line 5, in factory
node_class = load(MM_NODES_ENTRYPOINT, classname)
File "/opt/minemeld/engine/core/minemeld/loader.py", line 128, in load
return mmep.ep.load()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2302, in load
return self.resolve()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2308, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named YTExample
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/opt/minemeld/engine/core/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/core/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/core/minemeld/ft/__init__.py", line 5, in factory

...

At this point, this is just an exercise for me, and I think I'm going to punt.  Really, I'm interested in generating a process to download a feed of IP addresses.  Does that just involve making a prototype? 

Thanks...

[lmori]

Hi @vb0398,

what is the protocol and format of the IP address list you want to pull ? Chances you just a need a prototype for it.

Looking at you nodes.json you still have a dot instead of a colon in the minemeld.ft.ytexample.YTExample entrypoint between minemeld.ft.ytexample and YTExample, it should look like this:

"minemeld.ft.ytexample.YTExample": {
"class": "minemeld.ft.ytexample:YTExample"
}

after fixing this you should run:

sudo -u minemeld /opt/minemeld/engine/current/bin/pip install -e /opt/minemeled/engine/core/

[vb0398]

Hi @lmori,

    Got it...I thought I made the appropriate changes, but it looks like I didn't.  Now it works - thanks again..

Regarding what I'm really trying to do:

I'm downloading an IP list feed via HTTPS.  There are some comments at the top of the file, and then there's just an IP per line.

Perhaps something like the zeustracker prototype would be similar?  (i.e., https://github.com/PaloAltoNetworks/minemeld-node-prototypes/blob/master/prototypes/zeustracker.yml)

Best,

[lmori]

Hi @vb0398,

for a simple text file feed over HTTP you don't need a new class of Miner as there is already one implementing that protocol and format (minemeld.ft.http.HttpFT).

Suggestion, do this:

- in CONFIG click on the hamburger icon (bottom right) and search the prototype openbl.base

- click on the prototype and then click NEW, this will create a local copy of the prototype and you can change name and config

- in the config section of the new prototype modify the URL to point to your feed, the source_name, the confidence

- using the ignore_regex field you can specify regular expression to filter out the comments. The one in openbl.base will ignore all the lines starting with #

- you can then click OK to save the prototype and use it for a new Miner

If the engine does not start when you commit, just check the minemeld-engine.logs file to see the error in the prototype. You can then create a new version of the prototype with the fixes and use it for another Miner.

luigi

[vb0398]

hi @lmori,

   Great - one last question.  What if I have a comma delimited file that I want to parse various fields out of.  For example:

#IP, date, category, ...

1.1.1., 2017-01-01, bot, ...

Thanks...

[lmori]

Hi @vb0398,

you could have success using minemeld.ft.csv.CSVFT class to parse the CSV file and skip the comment using the ignore_regex parameter. See the docs about the parameters accepted by that Miner class here:

https://github.com/PaloAltoNetworks/minemeld-core/blob/master/docs/nodeconfig.rst

And you can use bambenekconsulting.c2_ipmasterlist as a starting prototype for your experiments.

[vb0398]

hi @lmori,

    Thanks for all your help so far...one other question - if the feed you're downloading is gzipped, what is the appropriate way to gunzip the file for processing within minemeld?

thanks...

[lmori]

If the file is compressed by the HTTP Server on the fly (https://en.wikipedia.org/wiki/HTTP_compression) in gzip, the python library used by the CSV and HTTP Miner (that is python requests) should automatically take care of decompressing the file.

If instead the feed is contained in a gzip file you need a new Miner subclassing the HTTP or CSV Miner to decmpress gzip on the fly. This is possible and easy to do but it requires some coding.

[vb0398]

hi @lmori,

  Thanks for your reply.  Actually, the file is stored on the webserver gzipped, so I think I will need to code something myself to gunzip the file.

Is there an example somewhere I can look at for reference?