topic Re: How to configure a miner to pull from a generic API in General Topics

How to configure a miner to pull from a generic API

Retired Member — Fri, 02 Feb 2018 01:41:27 GMT

Is there currently a prototype miner that can be configured and used to pull from a generic API?

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

Thanks!

-Chris

Re: How to configure a miner to pull from a generic API

xhoms — Fri, 02 Feb 2018 18:04:46 GMT

Hi @Retired Member,

MineMeld can grab indicators from generic API provided that the following conditions are met:

HTTP/S based API
No or Basic Authentication (user + password)
Single transaction (one call retrieves the whole indicator list – no pagination)
Indicators are provided in plain, html, csv or json format.

If one of the conditions is not met, then a custom node (miner) must be coded.

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 18:54:23 GMT

I too wish to add a generic API.

HTTP/S based API (CHECK)
No or Basic Authentication (user + password) (CHECK)
Single transaction (one call retrieves the whole indicator list – no pagination) (CHECK)
Indicators are provided in plain, html, csv or json format. (CHECK)

what class would I use? I have tried several. I see where I can enter username/token but not sure where to add the actual url to grab json file.

THIS IS NOT WORKING: class: minemeld.ft.anomali.Intelligence

here is my config

description: >

Threat Intelligence

url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

prototypes:

blackwired:

author: Jason

development_status: EXPERIMENTAL

node_type: miner

indicator_types: [ URL, IPv4, ]

tags:

- ConfidenceHigh

- ConfidenceLow

- ConfidenceMedium

- ShareLevelRed

description: >

Miner for careI. You need a valid API Key

to use this Miner.

config:

age_out:

default: 90d

sudden_death: true

interval: 3307

attributes:

share_level: red

confidence: 30

Re: How to configure a miner to pull from a generic API

xhoms — Tue, 06 Mar 2018 19:06:08 GMT

Hi @jsamide,

how does your content looks like?

If it looks like CSV then you need a Miner extending the minemeld.ft.csv.CSVFT class. Easiest way is by creating a prototype based on sslabusech.ipblacklist
If it looks like JSON then you need a Miner extending the minemeld.ft.json.SimpleJSON class. You can reach it by creating a prototype based on aws.AMAZON (educate yourself on JMESPath expressions - jmespath.org)
If it looks like Plain Text then you need a Miner extending the minemeld.ft.http.HttpFT class. Create a new prototype based on dshield.block for example.

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 19:08:55 GMT

@xhoms does the minemeld.ft.json.SimpleJSON class require a username/password?

Re: How to configure a miner to pull from a generic API

xhoms — Tue, 06 Mar 2018 19:10:40 GMT

@jsamide SimpleJSON supports username/password (basic auth) but it is not a requirement.

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 19:11:12 GMT

I will try that out now

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 19:17:13 GMT

getting Error in Commit: Bad request

my file:

description: >

Threat Intelligence

url: https://digital.wired.com

prototypes:

blackwired:

author: Sam

development_status: EXPERIMENTAL

node_type: miner

indicator_types: [ URL, IPv4, ]

tags:

- ConfidenceHigh

- ConfidenceLow

- ConfidenceMedium

- ShareLevelRed

description: >

Miner for careI. You need a valid API Key

to use this Miner.

config:

url: https://digital.black.com/exports/download/Palo-Alto-5a9ea59994e78.json

age_out:

default: 90d

sudden_death: true

interval: 3307

attributes:

share_level: red

confidence: 30

Re: How to configure a miner to pull from a generic API

xhoms — Tue, 06 Mar 2018 19:23:28 GMT

@jsamide, your miner configuration lacks class configuration parameters like extractor, indicator and fields.

I can help you with the class configuration (JMESPath expression indicator extractor) but you should share with us an example of the content that you want to mine.

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 19:26:34 GMT

I am trying to grab a json file that contains IPv and URL so would it look something like:

extractor: "badIP"

prefix: NOT SURE WHAT THIS POINTS TO

indicator: ip_prefix

fields:

- IP

- URL

Re: How to configure a miner to pull from a generic API

xhoms — Tue, 06 Mar 2018 19:29:36 GMT

@jsamide "extractor" should be a valid JMESPath expression that extracts a list of objects from your JSON content. "badIP" seems a too basic JMESPath expression. Have you tested the expression at http://jmespath.org/ ?

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 19:33:16 GMT

I will be doing some light reading

Re: How to configure a miner to pull from a generic API

jsamide — Tue, 06 Mar 2018 21:08:24 GMT

for the JMESPath where and how do I define the extractor process? Do I need to register this somewhere?

Re: How to configure a miner to pull from a generic API

xhoms — Fri, 09 Mar 2018 02:09:45 GMT

@jsamide,

imagine that your data looks like the following:

{
	"description": "list of indicators from foo.bar",
	"indicators": [
		{
			"type": "address",
			"data": "10.10.10.10",
			"source": "feed_x",
			"report_id": 188455
		},
		{
			"type": "address",
			"data": "11.11.11.11",
			"source": "feed_y",
			"report_id": 187411
		},
		{
			"type": "address",
			"data": "12.12.12.12",
			"source": "feed_z",
			"report_id": 677721
		}
	]
}

A valid value for the extractor configuration parameter for this case might be "indicators".

With such a value, the JMESPath engine inside the SimpleJSON miner will produce the following list:

[
  {
    "type": "address",
    "data": "10.10.10.10",
    "source": "feed_x",
    "report_id": 188455
  },
  {
    "type": "address",
    "data": "11.11.11.11",
    "source": "feed_y",
    "report_id": 187411
  },
  {
    "type": "address",
    "data": "12.12.12.12",
    "source": "feed_z",
    "report_id": 677721
  }
]

The indicator itself would be the value of the field "data". So, the value for the indicator configuration parameter should be "data".

And, finally, you might be interested in attaching the values of the fields "source" and "report_id" as metadata for the indicator. If you want to extract them, then assign the value "[source, report_id]" to the fields configuration parameter.

In summary: a valid configuration for the SimpleJSON for this case would be:

config
	extractor: indicators
	indicator: data
	fields:
		- source
		- report_id

Re: How to configure a miner to pull from a generic API

jsamide — Wed, 07 Mar 2018 17:56:52 GMT

so my data looks like this:

[{"IP":"69.213.8.8","URL":null},{"IP":"139.59.97.137","URL":null},{"IP":"192.99.142.235","URL":null},{"IP":"58.222.39.154","URL":null},{"IP":"69.64.147.10","URL":null},{"IP":"45.122.138.238","URL":null},]

here is my config:

config:

source_name: zero.IP

url: https://digital.wired.com/exports/download/Palo-Alto-5a9ea59994e78.json

Extractor: IP

prefix: sc

indicator: IP

fields:

-IP

-URL

now I am not able to find my file in the list of configurations

Re: How to configure a miner to pull from a generic API

jsamide — Wed, 07 Mar 2018 19:03:29 GMT

also, how do I add basic auth? is that an indicator? share level?

Re: How to configure a miner to pull from a generic API

xhoms — Wed, 07 Mar 2018 20:41:27 GMT

@jsamide,

for such a data source you should use the following configuration:

extractor = "[]"
indicator = "IP"
fields = ["URL"]

Re: How to configure a miner to pull from a generic API

jsamide — Wed, 07 Mar 2018 20:49:57 GMT

after many attempts I did figure this out with authentication.

thank you for ALL your help.

Re: How to configure a miner to pull from a generic API

Eddie_Brown — Tue, 18 Sep 2018 19:23:51 GMT

Hello I am attempting to create a miner using a paid threat intelligence providers API. The data deleivered is in a text format however the URL doesn't end in .txt. The URL does require basic authentication to view the data.

I have built my new prototype based off the dsheild.block prototype.

I have some questions regarding the authentication and the indicators and transform settings.

The API URL contains data in the below format with no headers above. just a giant list of text delimited with spaces and seperated into individual lines:

5.188.10.3 #Protection IP List: "hardcoded C2 for malicious downloader" Added 2018-03-14T22:49:12Z (59.939,30.3158) RU St Petersburg, Russia

Question 1: Is the basic authentication peice something I add into the prototype?

Question 2: I removed the following portions of the original dsheild.block
fields

I modified the indicator portion to only look for one IP address: regex: ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})

I modified the tranform to only list 1 value transform: \1

Does this look correct considering my data format?

Question 3: Their API does support a basic auth directly in the URL example: https://<api_username>:<api_password>@someurl.com/pan. I don't want to have my username and password in plain text within the prototype, how do I get around this?]

On a side note I have saveds this prototype and added the node. However, none of my indicators are being pulled. I'm sure I have screwed it up somewhere.

If you need any other information please let me know.

Thanks,

Eddie

Re: How to configure a miner to pull from a generic API

xhoms — Fri, 21 Sep 2018 19:34:22 GMT

Hi @Eddie_Brown

A1: Yes. Just use the "user:password@fqdn" notation

A2: Yes. The regex pattern you're using seems to match the content you're receiving

A3: You don't want these credentials to be stored in MineMeld? Then the only workaround I can think of is outsourcing them to an external API GW (AWS API GW in example) that could proxy the connection between MineMeld and the original feed. But you'll have just kicked your problem upstream.