How to configure a miner to pull from a generic API

Reply
Highlighted
L2 Linker

How to configure a miner to pull from a generic API

 

Is there currently a prototype miner that can be configured and used to pull from a generic API? 

 

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

 

Thanks!

-Chris

Tags (3)
L5 Sessionator

Re: How to configure a miner to pull from a generic API

Hi @chmotley,

 

MineMeld can grab indicators from generic API provided that the following conditions are met:

  • HTTP/S based API
  • No or Basic Authentication (user + password)
  • Single transaction (one call retrieves the whole indicator list – no pagination)
  • Indicators are provided in plain, html, csv or json format.

 

If one of the conditions is not met, then a custom node (miner) must be coded.

L2 Linker

Re: How to configure a miner to pull from a generic API

I too wish to add a generic API.  

 

  • HTTP/S based API  (CHECK)
  • No or Basic Authentication (user + password) (CHECK)
  • Single transaction (one call retrieves the whole indicator list – no pagination) (CHECK)
  • Indicators are provided in plain, html, csv or json format. (CHECK)

what class would I use?  I have tried several.  I see where I can enter username/token but not sure where to add the actual url to grab json file.

 

THIS IS NOT WORKING: class: minemeld.ft.anomali.Intelligence

 

here is my config

 

description: >

    Threat Intelligence

url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

prototypes:

    blackwired:

        author: Jason

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.anomali.Intelligence

        config:

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

L5 Sessionator

Re: How to configure a miner to pull from a generic API

Hi @jsamide,

 

how does your content looks like?

 

  • If it looks like CSV then you need a Miner extending the minemeld.ft.csv.CSVFT class. Easiest way is by creating a prototype based on sslabusech.ipblacklist
  • If it looks like JSON then you need a Miner extending the minemeld.ft.json.SimpleJSON class. You can reach it by creating a prototype based on aws.AMAZON (educate yourself on JMESPath expressions - jmespath.org)
  • If it looks like Plain Text then you need a Miner extending the minemeld.ft.http.HttpFT class. Create a new prototype based on dshield.block for example.

 

 

L2 Linker

Re: How to configure a miner to pull from a generic API

@xhoms does the minemeld.ft.json.SimpleJSON class require a username/password?

L5 Sessionator

Re: How to configure a miner to pull from a generic API

@jsamide SimpleJSON supports username/password (basic auth) but it is not a requirement.

L2 Linker

Re: How to configure a miner to pull from a generic API

I will try that out now

L2 Linker

Re: How to configure a miner to pull from a generic API

getting Error in Commit: Bad request

 

my file:

description: >

    Threat Intelligence

url: https://digital.wired.com

prototypes:

    blackwired:

        author: Sam

        development_status: EXPERIMENTAL

        node_type: miner

        indicator_types: [  URL, IPv4, ]

        tags:

            - ConfidenceHigh

            - ConfidenceLow

            - ConfidenceMedium

            - ShareLevelRed

        description: >

            Miner for careI. You need a valid API Key

            to use this Miner.

        class: minemeld.ft.json.SimpleJSON

        config:

    url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

            age_out:

                default: 90d

                sudden_death: true

                interval: 3307

            attributes:

                share_level: red

                confidence: 30

L5 Sessionator

Re: How to configure a miner to pull from a generic API

@jsamide, your miner configuration lacks class configuration parameters like extractor, indicator and fields.

 

I can help you with the class configuration (JMESPath expression indicator extractor) but you should share with us an example of the content that you want to mine.

 

L2 Linker

Re: How to configure a miner to pull from a generic API

I am trying to grab a json file that contains IPv and URL so would it look something like:

 

  extractor: "badIP"

            prefix: NOT SURE WHAT THIS POINTS TO

            indicator: ip_prefix

            fields:

                - IP

                - URL

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!