How to configure a miner to pull from a generic API

[chmotley]

Is there currently a prototype miner that can be configured and used to pull from a generic API? 

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

Thanks!

-Chris

[xhoms]

Hi @chmotley,

MineMeld can grab indicators from generic API provided that the following conditions are met:

• HTTP/S based API
• No or Basic Authentication (user + password)
• Single transaction (one call retrieves the whole indicator list – no pagination)
• Indicators are provided in plain, html, csv or json format.

If one of the conditions is not met, then a custom node (miner) must be coded.

[jsamide]

I too wish to add a generic API.  

• HTTP/S based API  (CHECK)
• No or Basic Authentication (user + password) (CHECK)
• Single transaction (one call retrieves the whole indicator list – no pagination) (CHECK)
• Indicators are provided in plain, html, csv or json format. (CHECK)

what class would I use?  I have tried several.  I see where I can enter username/token but not sure where to add the actual url to grab json file.

THIS IS NOT WORKING: class: minemeld.ft.anomali.Intelligence

here is my config

description: >

    Threat Intelligence

url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

prototypes:

    blackwired:

        author: Jason

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.anomali.Intelligence

        config:

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

[xhoms]

Hi @jsamide,

how does your content looks like?

• If it looks like CSV then you need a Miner extending the minemeld.ft.csv.CSVFT class. Easiest way is by creating a prototype based on sslabusech.ipblacklist
• If it looks like JSON then you need a Miner extending the minemeld.ft.json.SimpleJSON class. You can reach it by creating a prototype based on aws.AMAZON (educate yourself on JMESPath expressions - jmespath.org)
• If it looks like Plain Text then you need a Miner extending the minemeld.ft.http.HttpFT class. Create a new prototype based on dshield.block for example.

[jsamide]

@xhoms does the minemeld.ft.json.SimpleJSON class require a username/password?

[xhoms]

@jsamide SimpleJSON supports username/password (basic auth) but it is not a requirement.

[jsamide]

I will try that out now

[jsamide]

getting Error in Commit: Bad request

my file:

description: >

    Threat Intelligence

url: https://digital.wired.com

prototypes:

    blackwired:

        author: Sam

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.json.SimpleJSON

        config:

    url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

[xhoms]

@jsamide, your miner configuration lacks class configuration parameters like extractor, indicator and fields.

I can help you with the class configuration (JMESPath expression indicator extractor) but you should share with us an example of the content that you want to mine.

[jsamide]

I am trying to grab a json file that contains IPv and URL so would it look something like:

  extractor: "badIP"

            prefix: NOT SURE WHAT THIS POINTS TO

            indicator: ip_prefix

            fields:

                - IP

                - URL