How to configure a miner to pull from a generic API

[chmotley]

Is there currently a prototype miner that can be configured and used to pull from a generic API? 

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

Thanks!

-Chris

[jsamide]

also, how do I add basic auth?  is that an indicator? share level?

[xhoms]

@jsamide,

for such a data source you should use the following configuration:

• extractor = "[]"
• indicator = "IP"
• fields = ["URL"]

[jsamide]

after many attempts I did figure this out with authentication.

thank you for ALL your help. 

[Eddie_Brown]

Hello I am attempting to create a miner using a paid threat intelligence providers API. The data deleivered is in a text format however the URL doesn't end in .txt. The URL does require basic authentication to view the data.

I have built my new prototype based off the dsheild.block prototype. 

I have some questions regarding the authentication and the indicators and transform settings.

The API URL contains data in the below format with no headers above. just a giant list of text delimited with spaces and seperated into individual lines:

5.188.10.3 #Protection IP List: "hardcoded C2 for malicious downloader" Added 2018-03-14T22:49:12Z (59.939,30.3158) RU St Petersburg, Russia

Question 1: Is the basic authentication peice something I add into the prototype?

Question 2: I removed the following portions of the original dsheild.block 
fields 

I modified the indicator portion to only look for one IP address: regex: ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})

I modified the tranform to only list 1 value  transform: \1

Does this look correct considering my data format?

Question 3: Their API does support a basic auth directly in the URL example: https://<api_username>:<api_password>@someurl.com/pan. I don't want to have my username and password in plain text within the prototype, how do I get around this?]

On a side note I have saveds this prototype and added the node. However, none of my indicators are being pulled. I'm sure I have screwed it up somewhere.

If you need any other information please let me know.

Thanks,

Eddie

[xhoms]

Hi @Eddie_Brown

A1: Yes. Just use the "user:password@fqdn" notation

A2: Yes. The regex pattern you're using seems to match the content you're receiving

A3: You don't want these credentials to be stored in MineMeld? Then the only workaround I can think of is outsourcing them to an external API GW (AWS API GW in example) that could proxy the connection between MineMeld and the original feed. But you'll have just kicked your problem upstream.

[Eddie_Brown]

Thank you very much! It worked!