topic Re: how to write a simple miner documentation in General Topics

how to write a simple miner documentation

vb0398 — Mon, 15 May 2017 23:51:19 GMT

Hi there,

I'm a new user, so hopefully this is a simple question.

I installed minemeld via source code on ubuntu 14.04 using the instructions on this page :

https://github.com/PaloAltoNetworks/minemeld-ansible

The installation went smoothly and there were no errors.

I then went through the exercise of writing a test miner using these instructions : https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner

I create the ytexample.py file in the detailed directory, replaced /opt/minemeld/local/config/committed-config.yml with the node information available in the "How-To.." webpage, and restarted the minemeld service. From this point, I check the minemeld-engine.log file, and I see the following error:

minemeld-engine.log:2017-05-15T23:46:45 (14879)config._load_and_validate_config_from_file ERROR: Invalid config /opt/minemeld/local/config/committed-config.yml: Unknown node class minemeld.ft.ytexample.YTExample in testYT

Has anybody seen this error before?

Thanks...

Re: how to write a simple miner documentation

lmori — Wed, 17 May 2017 07:03:41 GMT

Hi @vb0398,

that guide should be updated, there are 2 additional steps:

- open the file nodes.json in the main directory of the minemeld engine and add the following to the dictionary of nodes:

[...]
},
"minemeld.ft.ytexample.YTExample": {"class": "minemeld.ft.ytexample.YTExample"}
}

- run "/opt/minemeld/engine/current/bin/pip install -e /opt/minemeld/engine/core"

NOTE: If you are looking into creating a new Miner my suggestion is to use external extensions, they are more flexible and agile. The same code of the Miner in the guide has been packaged as an extension here:

https://github.com/PaloAltoNetworks/youtube-miner

Re: how to write a simple miner documentation

vb0398 — Fri, 19 May 2017 07:26:20 GMT

Hi @lmori - Thanks for your reply.

I did those last 2 steps, and it still doesn't work. The error this time in the minemeld-engine.log :

ImportError: No module named YTExample
ImportError: No module named YTExample

I will go ahead and try the extension route.

Best,

Re: how to write a simple miner documentation

lmori — Fri, 19 May 2017 07:49:50 GMT

Hi @vb0398,

it seems that python is not able to find the YTExample module containing the Miner class.

Please could you attach the nodes.json file ? is the YTExample.py in minemeld/ft directory ?

Thanks,

luigi

Re: how to write a simple miner documentation

vb0398 — Fri, 19 May 2017 17:56:15 GMT

hi @lmori,

In your documentation, it says to name the file, 'ytexample.py' - all lowercase, and that is the name of the file in the "/opt/minemeld/engine/core/minemeld/ft" directory.

Attached the nodes.json file

Re: how to write a simple miner documentation

lmori — Thu, 25 May 2017 05:50:10 GMT

Hi @vb0398,

sorry for the late reply. There is a typo in you nodes.json file, the line should read:

[...]
    "minemeld.ft.ytexample.YTExample": {
        "class": "minemeld.ft.ytexample:YTExample"
    }
[...]

instead in your file you have:

[...]
    "minemeld.ft.ytexample.YTExample": {
        "class": "minemeld.ft.ytexample.YTExample"
    }
[...]

(":" is a Python thing)

Re: how to write a simple miner documentation

vb0398 — Mon, 29 May 2017 00:23:07 GMT

Ah - ok - corrected nodes.json - same error...

"class": "minemeld.ft.threatq:Export"
},
"minemeld.ft.tmt.DTIAPI": {
"class": "minemeld.ft.tmt:DTIAPI"
},
"minemeld.ft.vt.Notifications": {
"class": "minemeld.ft.vt:Notifications"
},
"minemeld.ft.mm.JSONSEQMiner": {
"class": "minemeld.ft.mm:JSONSEQMiner"
},
"minemeld.ft.ytexample.YTExample": {
"class": "minemeld.ft.ytexample.YTExample"
}
}

error:

...

2017-05-28T23:57:10 (6730)launcher.main INFO: multiprocessing: #cores: 1
2017-05-28T23:57:10 (6730)launcher.main INFO: multiprocessing: max #chassis: 1
2017-05-28T23:57:10 (6730)launcher.main INFO: Number of chassis: 1
2017-05-28T23:57:10 (6734)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ytexample.YTExample
2017-05-28T23:57:10 (6734)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/core/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/core/minemeld/ft/__init__.py", line 5, in factory
node_class = load(MM_NODES_ENTRYPOINT, classname)
File "/opt/minemeld/engine/core/minemeld/loader.py", line 128, in load
return mmep.ep.load()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2302, in load
return self.resolve()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2308, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named YTExample
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/opt/minemeld/engine/core/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/core/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/core/minemeld/ft/__init__.py", line 5, in factory

...

At this point, this is just an exercise for me, and I think I'm going to punt. Really, I'm interested in generating a process to download a feed of IP addresses. Does that just involve making a prototype?

Thanks...

Re: how to write a simple miner documentation

lmori — Mon, 29 May 2017 06:43:47 GMT

Hi @vb0398,

what is the protocol and format of the IP address list you want to pull ? Chances you just a need a prototype for it.

Looking at you nodes.json you still have a dot instead of a colon in the minemeld.ft.ytexample.YTExample entrypoint between minemeld.ft.ytexample and YTExample, it should look like this:

"minemeld.ft.ytexample.YTExample": {
"class": "minemeld.ft.ytexample:YTExample"
}

after fixing this you should run:

sudo -u minemeld /opt/minemeld/engine/current/bin/pip install -e /opt/minemeled/engine/core/

Re: how to write a simple miner documentation

vb0398 — Mon, 29 May 2017 17:54:39 GMT

Hi @lmori,

Got it...I thought I made the appropriate changes, but it looks like I didn't. Now it works - thanks again..

Regarding what I'm really trying to do:

I'm downloading an IP list feed via HTTPS. There are some comments at the top of the file, and then there's just an IP per line.

Perhaps something like the zeustracker prototype would be similar? (i.e., https://github.com/PaloAltoNetworks/minemeld-node-prototypes/blob/master/prototypes/zeustracker.yml)

Best,

Re: how to write a simple miner documentation

lmori — Mon, 29 May 2017 18:13:17 GMT

Hi @vb0398,

for a simple text file feed over HTTP you don't need a new class of Miner as there is already one implementing that protocol and format (minemeld.ft.http.HttpFT).

Suggestion, do this:

- in CONFIG click on the hamburger icon (bottom right) and search the prototype openbl.base

- click on the prototype and then click NEW, this will create a local copy of the prototype and you can change name and config

- in the config section of the new prototype modify the URL to point to your feed, the source_name, the confidence

- using the ignore_regex field you can specify regular expression to filter out the comments. The one in openbl.base will ignore all the lines starting with #

- you can then click OK to save the prototype and use it for a new Miner

If the engine does not start when you commit, just check the minemeld-engine.logs file to see the error in the prototype. You can then create a new version of the prototype with the fixes and use it for another Miner.

luigi

Re: how to write a simple miner documentation

vb0398 — Mon, 29 May 2017 19:18:38 GMT

hi @lmori,

Great - one last question. What if I have a comma delimited file that I want to parse various fields out of. For example:

#IP, date, category, ...

1.1.1., 2017-01-01, bot, ...

Thanks...

Re: how to write a simple miner documentation

lmori — Mon, 29 May 2017 19:43:36 GMT

Hi @vb0398,

you could have success using minemeld.ft.csv.CSVFT class to parse the CSV file and skip the comment using the ignore_regex parameter. See the docs about the parameters accepted by that Miner class here:

https://github.com/PaloAltoNetworks/minemeld-core/blob/master/docs/nodeconfig.rst

And you can use bambenekconsulting.c2_ipmasterlist as a starting prototype for your experiments.

Re: how to write a simple miner documentation

vb0398 — Tue, 30 May 2017 21:37:58 GMT

hi @lmori,

Thanks for all your help so far...one other question - if the feed you're downloading is gzipped, what is the appropriate way to gunzip the file for processing within minemeld?

thanks...

Re: how to write a simple miner documentation

lmori — Tue, 30 May 2017 21:59:52 GMT

If the file is compressed by the HTTP Server on the fly (https://en.wikipedia.org/wiki/HTTP_compression) in gzip, the python library used by the CSV and HTTP Miner (that is python requests) should automatically take care of decompressing the file.

If instead the feed is contained in a gzip file you need a new Miner subclassing the HTTP or CSV Miner to decmpress gzip on the fly. This is possible and easy to do but it requires some coding.

Re: how to write a simple miner documentation

vb0398 — Thu, 01 Jun 2017 21:49:13 GMT

hi @lmori,

Thanks for your reply. Actually, the file is stored on the webserver gzipped, so I think I will need to code something myself to gunzip the file.

Is there an example somewhere I can look at for reference?

Re: how to write a simple miner documentation

nbilal — Sun, 12 Nov 2017 22:15:55 GMT

@lmori

Thanks for the additional tips, it'd be great to get those in the documentation if possible. I mean these two additional steps:

that guide should be updated, there are 2 additional steps:

open the file nodes.json in the main directory of the minemeld engine and add the corresponding dictionary entry
run "/opt/minemeld/engine/current/bin/pip install -e /opt/minemeld/engine/core"

Actually, do you think we could get a guide on writing external extensions? Maybe it could replace the existing "write a simple miner" guide in the wiki.

I had the same issues in writing my miner (this one for Imperva's "Incapsula" cloud WAF public IP ranges), though after rebooting the VM it seems to have successfully updated everything and the miner is functional. I'm attaching the following files:

/opt/minemeld/engine/core/minemeld/ft/incapsula.py
/opt/minemeld/local/prototypes/incapsula.yml
/opt/minemeld/engine/core/nodes.json

I've looked at the youtube-miner repo but as a non-developer would find it a little helpful to get a high-level outline of the required structure for an external extension. It would be nice to be able to rewrite this standard miner as an extension.

Thanks again!

Nasir

Re: how to write a simple miner documentation

nbilal — Mon, 20 Nov 2017 15:44:59 GMT

Hey @lmori,

I've been trying to rewrite my incapsula miner as an external extension by parroting the youtube-miner example, but after installing it via the external extension menu under System > Extensions > Git and successfully activating it, I get the "COMMIT FAILED: Unknown node class minemeld.ft.incapsula.IPv4 in miner_incapsula_ipv4" in the web UI.

I am attaching my minemeld-engine.log, minemeld-web.log, and supervisor.log. Also, here is the link to the github repo containing the extension:

https://github.com/bilalbox/incapsula-miner

I'd be very appreciative of any pointers you could provide! I'm assuming there is some additional config required in my extension in order to force an update the local nodes.json in my minemeld VM?

Thanks,

Nasir

Re: how to write a simple miner documentation

xhoms — Mon, 20 Nov 2017 15:57:11 GMT

@nbilal : There are a couple of issues.

First, you're duplicating entry points in the minemeld.json file. The second entry should be "incapsulaminer.IPv6" instead of "incapsulaminer.node:IPv4".

Then, in the prototype file (incapsula.yml), you should reference these entry points (incapsulaminer.IPv4 and incapsulaminer.IPv6) instead of the non-existant ones minemeld.ft.incapsula.IPv4 and minemeld.ft.incapsula.IPv6

Re: how to write a simple miner documentation

nbilal — Mon, 20 Nov 2017 17:34:27 GMT

Thanks @xhoms. ...rookie mistakes! I also had to fix a bad import statement (minemeld.ft can be referenced as "." in a local miner, but the full path "minemeld.ft.x" must be given in the external extension).

We are good to go!

Thanks again for your support,

Nasir