how to write a simple miner documentation

Reply
L2 Linker

how to write a simple miner documentation

Hi there,

   I'm a new user, so hopefully this is a simple question.

 

I installed minemeld via source code on ubuntu 14.04 using the instructions on this page : 

https://github.com/PaloAltoNetworks/minemeld-ansible

 

 The installation went smoothly and there were no errors.

 

I then went through the exercise of writing a test miner using these instructions :  https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner

 

I create the ytexample.py file in the detailed directory, replaced /opt/minemeld/local/config/committed-config.yml with the node information available in the "How-To.." webpage, and restarted the minemeld service.  From this point, I check the minemeld-engine.log file, and I see the following error:

 

minemeld-engine.log:2017-05-15T23:46:45 (14879)config._load_and_validate_config_from_file ERROR: Invalid config /opt/minemeld/local/config/committed-config.yml: Unknown node class minemeld.ft.ytexample.YTExample in testYT

 

Has anybody seen this error before?

 

Thanks...

L7 Applicator

Re: how to write a simple miner documentation

Hi @vb0398,

that guide should be updated, there are 2 additional steps:

- open the file nodes.json in the main directory of the minemeld engine and add the following to the dictionary of nodes:

[...]
},
"minemeld.ft.ytexample.YTExample": {"class": "minemeld.ft.ytexample.YTExample"}
}

- run "/opt/minemeld/engine/current/bin/pip install -e /opt/minemeld/engine/core"

 

NOTE: If you are looking into creating a new Miner my suggestion is to use external extensions, they are more flexible and agile. The same code of the Miner in the guide has been packaged as an extension here:

https://github.com/PaloAltoNetworks/youtube-miner

L2 Linker

Re: how to write a simple miner documentation

Hi @lmori - Thanks for your reply.

 

I did those last 2 steps, and it still doesn't work.  The error this time in the minemeld-engine.log :

 

ImportError: No module named YTExample
ImportError: No module named YTExample

 

I will go ahead and try the extension route.

 

Best,

 

L7 Applicator

Re: how to write a simple miner documentation

Hi @vb0398,

it seems that python is not able to find the YTExample module containing the Miner class.

Please could you attach the nodes.json file ? is the YTExample.py in minemeld/ft directory ?

 

Thanks,

luigi

L2 Linker

Re: how to write a simple miner documentation

hi @lmori,

   In your documentation, it says to name the file, 'ytexample.py' - all lowercase, and that is the name of the file in the "/opt/minemeld/engine/core/minemeld/ft" directory.  

 

Attached the nodes.json file

 

 

L7 Applicator

Re: how to write a simple miner documentation

Hi @vb0398,

sorry for the late reply. There is a typo in you nodes.json file, the line should read:

[...]
    "minemeld.ft.ytexample.YTExample": {
        "class": "minemeld.ft.ytexample:YTExample"
    }
[...]

instead in your file you have:

[...]
    "minemeld.ft.ytexample.YTExample": {
        "class": "minemeld.ft.ytexample.YTExample"
    }
[...]

(":" is a Python thing)

L2 Linker

Re: how to write a simple miner documentation

Ah - ok - corrected nodes.json - same error...

 

  "class": "minemeld.ft.threatq:Export"
},
"minemeld.ft.tmt.DTIAPI": {
"class": "minemeld.ft.tmt:DTIAPI"
},
"minemeld.ft.vt.Notifications": {
"class": "minemeld.ft.vt:Notifications"
},
"minemeld.ft.mm.JSONSEQMiner": {
"class": "minemeld.ft.mm:JSONSEQMiner"
},
"minemeld.ft.ytexample.YTExample": {
"class": "minemeld.ft.ytexample.YTExample"
}
}

 

error:

 

 

 

...

2017-05-28T23:57:10 (6730)launcher.main INFO: multiprocessing: #cores: 1
2017-05-28T23:57:10 (6730)launcher.main INFO: multiprocessing: max #chassis: 1
2017-05-28T23:57:10 (6730)launcher.main INFO: Number of chassis: 1
2017-05-28T23:57:10 (6734)loader.load INFO: Loading minemeld_nodes:minemeld.ft.ytexample.YTExample
2017-05-28T23:57:10 (6734)launcher._run_chassis ERROR: Exception in chassis main procedure
Traceback (most recent call last):
File "/opt/minemeld/engine/core/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/core/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/core/minemeld/ft/__init__.py", line 5, in factory
node_class = load(MM_NODES_ENTRYPOINT, classname)
File "/opt/minemeld/engine/core/minemeld/loader.py", line 128, in load
return mmep.ep.load()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2302, in load
return self.resolve()
File "/opt/minemeld/engine/current/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2308, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named YTExample
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/opt/minemeld/engine/core/minemeld/run/launcher.py", line 53, in _run_chassis
c.configure(fts)
File "/opt/minemeld/engine/core/minemeld/chassis.py", line 102, in configure
config=ftconfig.get('config', {})
File "/opt/minemeld/engine/core/minemeld/ft/__init__.py", line 5, in factory

...

 

At this point, this is just an exercise for me, and I think I'm going to punt.  Really, I'm interested in generating a process to download a feed of IP addresses.  Does that just involve making a prototype? 

 

Thanks...

 

 

 

L7 Applicator

Re: how to write a simple miner documentation

Hi @vb0398,

what is the protocol and format of the IP address list you want to pull ? Chances you just a need a prototype for it.

 

Looking at you nodes.json you still have a dot instead of a colon in the minemeld.ft.ytexample.YTExample entrypoint between minemeld.ft.ytexample and YTExample, it should look like this:

"minemeld.ft.ytexample.YTExample": {
"class": "minemeld.ft.ytexample:YTExample"
}

after fixing this you should run:

sudo -u minemeld /opt/minemeld/engine/current/bin/pip install -e /opt/minemeled/engine/core/
L2 Linker

Re: how to write a simple miner documentation

Hi @lmori,

    Got it...I thought I made the appropriate changes, but it looks like I didn't.  Now it works - thanks again..

 

Regarding what I'm really trying to do:

 

I'm downloading an IP list feed via HTTPS.  There are some comments at the top of the file, and then there's just an IP per line.

 

Perhaps something like the zeustracker prototype would be similar?  (i.e., https://github.com/PaloAltoNetworks/minemeld-node-prototypes/blob/master/prototypes/zeustracker.yml)

 

Best,

 

L7 Applicator

Re: how to write a simple miner documentation

Hi @vb0398,

for a simple text file feed over HTTP you don't need a new class of Miner as there is already one implementing that protocol and format (minemeld.ft.http.HttpFT).

Suggestion, do this:

- in CONFIG click on the hamburger icon (bottom right) and search the prototype openbl.base

- click on the prototype and then click NEW, this will create a local copy of the prototype and you can change name and config

- in the config section of the new prototype modify the URL to point to your feed, the source_name, the confidence

- using the ignore_regex field you can specify regular expression to filter out the comments. The one in openbl.base will ignore all the lines starting with #

- you can then click OK to save the prototype and use it for a new Miner

 

If the engine does not start when you commit, just check the minemeld-engine.logs file to see the error in the prototype. You can then create a new version of the prototype with the fixes and use it for another Miner.

 

luigi

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!