How to configure a miner to pull from a generic API

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

How to configure a miner to pull from a generic API

L2 Linker

 

Is there currently a prototype miner that can be configured and used to pull from a generic API? 

 

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

 

Thanks!

-Chris

20 REPLIES 20

L5 Sessionator

Hi @chmotley,

 

MineMeld can grab indicators from generic API provided that the following conditions are met:

  • HTTP/S based API
  • No or Basic Authentication (user + password)
  • Single transaction (one call retrieves the whole indicator list – no pagination)
  • Indicators are provided in plain, html, csv or json format.

 

If one of the conditions is not met, then a custom node (miner) must be coded.

I too wish to add a generic API.  

 

  • HTTP/S based API  (CHECK)
  • No or Basic Authentication (user + password) (CHECK)
  • Single transaction (one call retrieves the whole indicator list – no pagination) (CHECK)
  • Indicators are provided in plain, html, csv or json format. (CHECK)

what class would I use?  I have tried several.  I see where I can enter username/token but not sure where to add the actual url to grab json file.

 

THIS IS NOT WORKING: class: minemeld.ft.anomali.Intelligence

 

here is my config

 

description: >

    Threat Intelligence

url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

prototypes:

    blackwired:

        author: Jason

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.anomali.Intelligence

        config:

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

Hi @jsamide,

 

how does your content looks like?

 

  • If it looks like CSV then you need a Miner extending the minemeld.ft.csv.CSVFT class. Easiest way is by creating a prototype based on sslabusech.ipblacklist
  • If it looks like JSON then you need a Miner extending the minemeld.ft.json.SimpleJSON class. You can reach it by creating a prototype based on aws.AMAZON (educate yourself on JMESPath expressions - jmespath.org)
  • If it looks like Plain Text then you need a Miner extending the minemeld.ft.http.HttpFT class. Create a new prototype based on dshield.block for example.

 

 

@xhoms does the minemeld.ft.json.SimpleJSON class require a username/password?

@jsamide SimpleJSON supports username/password (basic auth) but it is not a requirement.

I will try that out now

getting Error in Commit: Bad request

 

my file:

description: >

    Threat Intelligence

url: https://digital.wired.com

prototypes:

    blackwired:

        author: Sam

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.json.SimpleJSON

        config:

    url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

@jsamide, your miner configuration lacks class configuration parameters like extractor, indicator and fields.

 

I can help you with the class configuration (JMESPath expression indicator extractor) but you should share with us an example of the content that you want to mine.

 

I am trying to grab a json file that contains IPv and URL so would it look something like:

 

  extractor: "badIP"

            prefix: NOT SURE WHAT THIS POINTS TO

            indicator: ip_prefix

            fields:

                - IP

                - URL

@jsamide "extractor" should be a valid JMESPath expression that extracts a list of objects from your JSON content. "badIP" seems a too basic JMESPath expression. Have you tested the expression at http://jmespath.org/ ?

I will be doing some light reading

for the JMESPath where and how do I define the extractor process?  Do I need to register this somewhere? 

@jsamide,

 

imagine that your data looks like the following:

{
	"description": "list of indicators from foo.bar",
	"indicators": [
		{
			"type": "address",
			"data": "10.10.10.10",
			"source": "feed_x",
			"report_id": 188455
		},
		{
			"type": "address",
			"data": "11.11.11.11",
			"source": "feed_y",
			"report_id": 187411
		},
		{
			"type": "address",
			"data": "12.12.12.12",
			"source": "feed_z",
			"report_id": 677721
		}
	]
}

A valid value for the extractor configuration parameter for this case might be "indicators".

 

With such a value, the JMESPath engine inside the SimpleJSON miner will produce the following list:

[
  {
    "type": "address",
    "data": "10.10.10.10",
    "source": "feed_x",
    "report_id": 188455
  },
  {
    "type": "address",
    "data": "11.11.11.11",
    "source": "feed_y",
    "report_id": 187411
  },
  {
    "type": "address",
    "data": "12.12.12.12",
    "source": "feed_z",
    "report_id": 677721
  }
]

The indicator itself would be the value of the field "data". So, the value for the indicator configuration parameter should be "data".

 

And, finally, you might be interested in attaching the values of the fields "source" and "report_id" as metadata for the indicator. If you want to extract them, then assign the value "[source, report_id]" to the fields configuration parameter.

 

In summary: a valid configuration for the SimpleJSON for this case would be:

config
	extractor: indicators
	indicator: data
	fields:
		- source
		- report_id

 

so my data looks like this:

[{"IP":"69.213.8.8","URL":null},{"IP":"139.59.97.137","URL":null},{"IP":"192.99.142.235","URL":null},{"IP":"58.222.39.154","URL":null},{"IP":"69.64.147.10","URL":null},{"IP":"45.122.138.238","URL":null},]

 

here is my config:

 

        class: minemeld.ft.json.SimpleJSON

        config:

            source_name: zero.IP

            url: https://digital.wired.com/exports/download/Palo-Alto-5a9ea59994e78.json

            Extractor: IP

            prefix: sc

            indicator: IP

            fields:

                -IP

                -URL

 

now I am not able to find my file in the list of configurations

  • 15067 Views
  • 20 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!