Automatization of Minemeld with API

[AlexForeroITS]

Hi,

Im trying to create node using "autofocus.sampleMiner" proto using API.

When I try to get existing node config I get the following answer when I run /config/node/N

HTTP/2 200
server: nginx/1.11.3
date: Wed, 02 Jan 2019 12:19:16 GMT
content-type: application/json
content-length: 227
expires: Wed, 02 Jan 2019 12:19:15 GMT
cache-control: no-cache
strict-transport-security: max-age=15724800; includeSubDomains; preload

{
  "result": {
    "name": "af-miner",
    "properties": {
      "inputs": [],
      "output": true,
      "prototype": "autofocus.samplesMiner"
    },
    "version": "e5c8dd3c-16b5-4dbf-9798-72f8XXXXXXX"
  }
}

I would like to customize "autofocus query" parameter in order to create custom miners. Any option?

Thank you in advance,

Happy new year.

[ITSCERT]

Hi,

I just ran this:

{
"name": "My af miner",
"properties": {
"prototype": "autofocus.samplesMiner",

"config": {
"attributes": {
"share_level": "red",
"confidence": "100"

}
},
"output": true,
"query": {"operator":"all","children":[{"field":"sample.malware","operator":"is","value":1}]}
},
"version": "e5c8dd3c-16b5-4dbf-9798-XXXXXXXXXXX"
}

And the node has been created sucessfully but now, I need to set a specific query, scope, and antifacts. Where should I put them?

Thank you in advance,

[lmori]

Hi @ITSCERT,

there is a dedicated API to add what is called "side config", i.e. parameters read by the node at runtime. The query should be added using that API.

Example:

PUT https://<minemeld>/config/data/<miner-name>_side_config

{
"query":"{\"operator\":\"all\",\"children\":[{\"field\":\"sample.malware\",\"operator\":\"is\",\"value\":1},{\"field\":\"sample.create_date\",\"operator\":\"is after\",\"value\":[\"2019-01-07\",\"2019-01-07\"]}]}",
"scope":"global",
"artifact_source":"af"
}

[ITSCERT]

Hi,

I followed steps you detailed to me:

1) CreateNode running following cmds

query = '{ "name": "My_af_miner2", "properties": { "prototype": "autofocus.samplesMiner", "config": { "attributes": { "share_level": "red", "confidence": "100" } }, "output": true }, "version": "9c3d9621xxxxxxxxx."
        createNode = requests.post(URL+URI_STATUS, verify=False, auth=HTTPBasicAuth('Y, 'X'), headers={ "Content-Type": "application/json" }, data=query)
        resp = createNode.json()

The result is the following:

{'result': {'id': 87, 'version': '9c3d9621XXXXX0b-86ac-6XXXaa+0'}}

After create node, I verify that is correct on minemeld gui side. The only thing that needs are side_config params.

I run as you told me the following:

URI_STATUS='/config/data/My_af_miner2_side_config'
query = '{ "query":"{\"operator\":\"all\",\"children\":[{\"field\":\"sample.malware\",\"operator\":\"is\",\"value\":1},{\"field\":\"sample.create_date\",\"operator\":\"is after\",\"value\":[\"2019-01-07\",\"2019-01-07\"]}]}", "scope":"global", "artifact_source":"af" }'
modifyNode = requests.put(URL+URI_STATUS, verify=False, auth=HTTPBasicAuth('y', 'x'), headers={ "Content-Type": "application/json" }, data=query)
resp = modifyNode.json()
print(resp)

And the answer is:

{'result': 'ok'}

But the config of the minner has not changed.

I have tried also to restart engine, but i got backoff error. Any other suggestions?

Thank you in advance,

[ITSCERT]

hi @lmori,

After PUT what you told me to test, I receive ok, but the query is not set. If I restart engine, it fails and i need to delete node and commit  to make it working.

Thank you in advance,

[lmori]

Hi @ITSCERT,

I think you have some issues with quoting in the python code. If you want to pass a string with the JSON data encoded, it should look like (note the \\):

'{"query": "{\\"operator\\":\\"all\\",\\"children\\":[{\\"field\\":\\"sample.malware\\",\\"operator\\":\\"is\\",\\"value\\":1},{\\"field\\":\\"sample.create_date\\",\\"operator\\":\\"is after\\",\\"value\\":[\\"2019-01-07\\",\\"2019-01-07\\"]}]}", "artifact_source": "af", "scope": "global"}'

 Otherwise you can also pass the dictionary to requests and requests will encode it in json for you:

query = {u'query': u'{"operator":"all","children":[{"field":"sample.malware","operator":"is","value":1},{"field":"sample.create_date","operator":"is after","value":["2019-01-07","2019-01-07"]}]}', u'artifact_source': u'af', u'scope': u'global'}

[ITSCERT]

Solved!

Thank you @lmori