How to configure a miner to pull from a generic API

Reply
Highlighted
L5 Sessionator

Re: How to configure a miner to pull from a generic API

@jsamide "extractor" should be a valid JMESPath expression that extracts a list of objects from your JSON content. "badIP" seems a too basic JMESPath expression. Have you tested the expression at http://jmespath.org/ ?

Highlighted
L2 Linker

Re: How to configure a miner to pull from a generic API

I will be doing some light reading

Highlighted
L2 Linker

Re: How to configure a miner to pull from a generic API

for the JMESPath where and how do I define the extractor process?  Do I need to register this somewhere? 

Highlighted
L5 Sessionator

Re: How to configure a miner to pull from a generic API

@jsamide,

 

imagine that your data looks like the following:

{
	"description": "list of indicators from foo.bar",
	"indicators": [
		{
			"type": "address",
			"data": "10.10.10.10",
			"source": "feed_x",
			"report_id": 188455
		},
		{
			"type": "address",
			"data": "11.11.11.11",
			"source": "feed_y",
			"report_id": 187411
		},
		{
			"type": "address",
			"data": "12.12.12.12",
			"source": "feed_z",
			"report_id": 677721
		}
	]
}

A valid value for the extractor configuration parameter for this case might be "indicators".

 

With such a value, the JMESPath engine inside the SimpleJSON miner will produce the following list:

[
  {
    "type": "address",
    "data": "10.10.10.10",
    "source": "feed_x",
    "report_id": 188455
  },
  {
    "type": "address",
    "data": "11.11.11.11",
    "source": "feed_y",
    "report_id": 187411
  },
  {
    "type": "address",
    "data": "12.12.12.12",
    "source": "feed_z",
    "report_id": 677721
  }
]

The indicator itself would be the value of the field "data". So, the value for the indicator configuration parameter should be "data".

 

And, finally, you might be interested in attaching the values of the fields "source" and "report_id" as metadata for the indicator. If you want to extract them, then assign the value "[source, report_id]" to the fields configuration parameter.

 

In summary: a valid configuration for the SimpleJSON for this case would be:

config
	extractor: indicators
	indicator: data
	fields:
		- source
		- report_id

 

Highlighted
L2 Linker

Re: How to configure a miner to pull from a generic API

so my data looks like this:

[{"IP":"69.213.8.8","URL":null},{"IP":"139.59.97.137","URL":null},{"IP":"192.99.142.235","URL":null},{"IP":"58.222.39.154","URL":null},{"IP":"69.64.147.10","URL":null},{"IP":"45.122.138.238","URL":null},]

 

here is my config:

 

        class: minemeld.ft.json.SimpleJSON

        config:

            source_name: zero.IP

            url: https://digital.wired.com/exports/download/Palo-Alto-5a9ea59994e78.json

            Extractor: IP

            prefix: sc

            indicator: IP

            fields:

                -IP

                -URL

 

now I am not able to find my file in the list of configurations

Highlighted
L2 Linker

Re: How to configure a miner to pull from a generic API

also, how do I add basic auth?  is that an indicator? share level?

Highlighted
L5 Sessionator

Re: How to configure a miner to pull from a generic API

@jsamide,

 

for such a data source you should use the following configuration:

  • extractor = "[]"
  • indicator = "IP"
  • fields = ["URL"]
Highlighted
L2 Linker

Re: How to configure a miner to pull from a generic API

after many attempts I did figure this out with authentication.

 

thank you for ALL your help. 

Highlighted
L2 Linker

Re: How to configure a miner to pull from a generic API

Hello I am attempting to create a miner using a paid threat intelligence providers API. The data deleivered is in a text format however the URL doesn't end in .txt. The URL does require basic authentication to view the data.

I have built my new prototype based off the dsheild.block prototype. 

I have some questions regarding the authentication and the indicators and transform settings.

The API URL contains data in the below format with no headers above. just a giant list of text delimited with spaces and seperated into individual lines:

5.188.10.3 #Protection IP List: "hardcoded C2 for malicious downloader" Added 2018-03-14T22:49:12Z (59.939,30.3158) RU St Petersburg, Russia

Question 1: Is the basic authentication peice something I add into the prototype?

Question 2: I removed the following portions of the original dsheild.block 
fields 

I modified the indicator portion to only look for one IP address: regex: ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})

I modified the tranform to only list 1 value  transform: \1

Does this look correct considering my data format?

Question 3: Their API does support a basic auth directly in the URL example: https://<api_username>:<api_password>@someurl.com/pan. I don't want to have my username and password in plain text within the prototype, how do I get around this?]

 

On a side note I have saveds this prototype and added the node. However, none of my indicators are being pulled. I'm sure I have screwed it up somewhere.

 

If you need any other information please let me know.

 

Thanks,

Eddie

Highlighted
L5 Sessionator

Re: How to configure a miner to pull from a generic API

Hi @Eddie_Brown

 

A1: Yes. Just use the "user:password@fqdn" notation

A2: Yes. The regex pattern you're using seems to match the content you're receiving

A3: You don't want these credentials to be stored in MineMeld? Then the only workaround I can think of is outsourcing them to an external API GW (AWS API GW in example) that could proxy the connection between MineMeld and the original feed. But you'll have just kicked your problem upstream.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!