how to write a simple miner documentation

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

how to write a simple miner documentation

L2 Linker

Hi there,

   I'm a new user, so hopefully this is a simple question.

 

I installed minemeld via source code on ubuntu 14.04 using the instructions on this page : 

https://github.com/PaloAltoNetworks/minemeld-ansible

 

 The installation went smoothly and there were no errors.

 

I then went through the exercise of writing a test miner using these instructions :  https://github.com/PaloAltoNetworks/minemeld/wiki/How-To-Write-a-Simple-Miner

 

I create the ytexample.py file in the detailed directory, replaced /opt/minemeld/local/config/committed-config.yml with the node information available in the "How-To.." webpage, and restarted the minemeld service.  From this point, I check the minemeld-engine.log file, and I see the following error:

 

minemeld-engine.log:2017-05-15T23:46:45 (14879)config._load_and_validate_config_from_file ERROR: Invalid config /opt/minemeld/local/config/committed-config.yml: Unknown node class minemeld.ft.ytexample.YTExample in testYT

 

Has anybody seen this error before?

 

Thanks...

18 REPLIES 18

@lmori

 

Thanks for the additional tips, it'd be great to get those in the documentation if possible. I mean these two additional steps:

that guide should be updated, there are 2 additional steps:

  1. open the file nodes.json in the main directory of the minemeld engine and add the corresponding dictionary entry
  2. run "/opt/minemeld/engine/current/bin/pip install -e /opt/minemeld/engine/core"

Actually, do you think we could get a guide on writing external extensions? Maybe it could replace the existing "write a simple miner" guide in the wiki.

 

I had the same issues in writing my miner (this one for Imperva's "Incapsula" cloud WAF public IP ranges), though after rebooting the VM it seems to have successfully updated everything and the miner is functional. I'm attaching the following files:

/opt/minemeld/engine/core/minemeld/ft/incapsula.py
/opt/minemeld/local/prototypes/incapsula.yml
/opt/minemeld/engine/core/nodes.json

 

I've looked at the youtube-miner repo but as a non-developer would find it a little helpful to get a high-level outline of the required structure for an external extension. It would be nice to be able to rewrite this standard miner as an extension.

 

Thanks again!

Nasir

Hey @lmori,

 

I've been trying to rewrite my incapsula miner as an external extension by parroting the youtube-miner example, but after installing it via the external extension menu under System > Extensions > Git and successfully activating it, I get the "COMMIT FAILED: Unknown node class minemeld.ft.incapsula.IPv4 in miner_incapsula_ipv4" in the web UI.

 

I am attaching my minemeld-engine.log, minemeld-web.log, and supervisor.log. Also, here is the link to the github repo containing the extension: 

https://github.com/bilalbox/incapsula-miner

 

I'd be very appreciative of any pointers you could provide! I'm assuming there is some additional config required in my extension in order to force an update the local nodes.json in my minemeld VM?

 

Thanks,

Nasir

@nbilal : There are a couple of issues.

 

First, you're duplicating entry points in the minemeld.json file. The second entry should be "incapsulaminer.IPv6" instead of "incapsulaminer.node:IPv4".

 

Then, in the prototype file (incapsula.yml), you should reference these entry points (incapsulaminer.IPv4 and incapsulaminer.IPv6) instead of the non-existant ones minemeld.ft.incapsula.IPv4 and minemeld.ft.incapsula.IPv6

 

Thanks @xhoms. ...rookie mistakes! I also had to fix a bad import statement (minemeld.ft can be referenced as "." in a local miner, but the full path "minemeld.ft.x" must be given in the external extension).

 

 

We are good to go!

 

Thanks again for your support,

Nasir

  • 11472 Views
  • 18 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!