How to configure a miner to pull from a generic API

[chmotley]

Is there currently a prototype miner that can be configured and used to pull from a generic API? 

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

Thanks!

-Chris

[xhoms]

Hi @chmotley,

MineMeld can grab indicators from generic API provided that the following conditions are met:

• HTTP/S based API
• No or Basic Authentication (user + password)
• Single transaction (one call retrieves the whole indicator list – no pagination)
• Indicators are provided in plain, html, csv or json format.

If one of the conditions is not met, then a custom node (miner) must be coded.

[jsamide]

I too wish to add a generic API.  

• HTTP/S based API  (CHECK)
• No or Basic Authentication (user + password) (CHECK)
• Single transaction (one call retrieves the whole indicator list – no pagination) (CHECK)
• Indicators are provided in plain, html, csv or json format. (CHECK)

what class would I use?  I have tried several.  I see where I can enter username/token but not sure where to add the actual url to grab json file.

THIS IS NOT WORKING: class: minemeld.ft.anomali.Intelligence

here is my config

description: >

    Threat Intelligence

url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

prototypes:

    blackwired:

        author: Jason

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.anomali.Intelligence

        config:

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

[xhoms]

Hi @jsamide,

how does your content looks like?

• If it looks like CSV then you need a Miner extending the minemeld.ft.csv.CSVFT class. Easiest way is by creating a prototype based on sslabusech.ipblacklist
• If it looks like JSON then you need a Miner extending the minemeld.ft.json.SimpleJSON class. You can reach it by creating a prototype based on aws.AMAZON (educate yourself on JMESPath expressions - jmespath.org)
• If it looks like Plain Text then you need a Miner extending the minemeld.ft.http.HttpFT class. Create a new prototype based on dshield.block for example.

[jsamide]

@xhoms does the minemeld.ft.json.SimpleJSON class require a username/password?

[xhoms]

@jsamide SimpleJSON supports username/password (basic auth) but it is not a requirement.

[jsamide]

I will try that out now

[jsamide]

getting Error in Commit: Bad request

my file:

description: >

    Threat Intelligence

url: https://digital.wired.com

prototypes:

    blackwired:

        author: Sam

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.json.SimpleJSON

        config:

    url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

[xhoms]

@jsamide, your miner configuration lacks class configuration parameters like extractor, indicator and fields.

I can help you with the class configuration (JMESPath expression indicator extractor) but you should share with us an example of the content that you want to mine.

[jsamide]

I am trying to grab a json file that contains IPv and URL so would it look something like:

  extractor: "badIP"

            prefix: NOT SURE WHAT THIS POINTS TO

            indicator: ip_prefix

            fields:

                - IP

                - URL

[xhoms]

@jsamide "extractor" should be a valid JMESPath expression that extracts a list of objects from your JSON content. "badIP" seems a too basic JMESPath expression. Have you tested the expression at http://jmespath.org/ ?

[jsamide]

I will be doing some light reading

[jsamide]

for the JMESPath where and how do I define the extractor process?  Do I need to register this somewhere? 

[xhoms]

@jsamide,

imagine that your data looks like the following:

{
	"description": "list of indicators from foo.bar",
	"indicators": [
		{
			"type": "address",
			"data": "10.10.10.10",
			"source": "feed_x",
			"report_id": 188455
		},
		{
			"type": "address",
			"data": "11.11.11.11",
			"source": "feed_y",
			"report_id": 187411
		},
		{
			"type": "address",
			"data": "12.12.12.12",
			"source": "feed_z",
			"report_id": 677721
		}
	]
}

A valid value for the extractor configuration parameter for this case might be "indicators".

With such a value, the JMESPath engine inside the SimpleJSON miner will produce the following list:

[
  {
    "type": "address",
    "data": "10.10.10.10",
    "source": "feed_x",
    "report_id": 188455
  },
  {
    "type": "address",
    "data": "11.11.11.11",
    "source": "feed_y",
    "report_id": 187411
  },
  {
    "type": "address",
    "data": "12.12.12.12",
    "source": "feed_z",
    "report_id": 677721
  }
]

The indicator itself would be the value of the field "data". So, the value for the indicator configuration parameter should be "data".

And, finally, you might be interested in attaching the values of the fields "source" and "report_id" as metadata for the indicator. If you want to extract them, then assign the value "[source, report_id]" to the fields configuration parameter.

In summary: a valid configuration for the SimpleJSON for this case would be:

config
	extractor: indicators
	indicator: data
	fields:
		- source
		- report_id

[jsamide]

so my data looks like this:

[{"IP":"69.213.8.8","URL":null},{"IP":"139.59.97.137","URL":null},{"IP":"192.99.142.235","URL":null},{"IP":"58.222.39.154","URL":null},{"IP":"69.64.147.10","URL":null},{"IP":"45.122.138.238","URL":null},]

here is my config:

        class: minemeld.ft.json.SimpleJSON

        config:

            source_name: zero.IP

            url: https://digital.wired.com/exports/download/Palo-Alto-5a9ea59994e78.json

            Extractor: IP

            prefix: sc

            indicator: IP

            fields:

                -IP

                -URL

now I am not able to find my file in the list of configurations