How to configure a miner to pull from a generic API

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

How to configure a miner to pull from a generic API

L2 Linker

 

Is there currently a prototype miner that can be configured and used to pull from a generic API? 

 

My example is Infoblox, but I can see this working with multiple infrastructure tools. I'm working with both AutoFocus-hosted Minemeld, and the stand-alone VM.

 

Thanks!

-Chris

20 REPLIES 20

also, how do I add basic auth?  is that an indicator? share level?

@jsamide,

 

for such a data source you should use the following configuration:

  • extractor = "[]"
  • indicator = "IP"
  • fields = ["URL"]

after many attempts I did figure this out with authentication.

 

thank you for ALL your help. 

Hello I am attempting to create a miner using a paid threat intelligence providers API. The data deleivered is in a text format however the URL doesn't end in .txt. The URL does require basic authentication to view the data.

I have built my new prototype based off the dsheild.block prototype. 

I have some questions regarding the authentication and the indicators and transform settings.

The API URL contains data in the below format with no headers above. just a giant list of text delimited with spaces and seperated into individual lines:

5.188.10.3 #Protection IP List: "hardcoded C2 for malicious downloader" Added 2018-03-14T22:49:12Z (59.939,30.3158) RU St Petersburg, Russia

Question 1: Is the basic authentication peice something I add into the prototype?

Question 2: I removed the following portions of the original dsheild.block 
fields 

I modified the indicator portion to only look for one IP address: regex: ^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})

I modified the tranform to only list 1 value  transform: \1

Does this look correct considering my data format?

Question 3: Their API does support a basic auth directly in the URL example: https://<api_username>:<api_password>@someurl.com/pan. I don't want to have my username and password in plain text within the prototype, how do I get around this?]

 

On a side note I have saveds this prototype and added the node. However, none of my indicators are being pulled. I'm sure I have screwed it up somewhere.

 

If you need any other information please let me know.

 

Thanks,

Eddie

Hi @Eddie_Brown

 

A1: Yes. Just use the "user:password@fqdn" notation

A2: Yes. The regex pattern you're using seems to match the content you're receiving

A3: You don't want these credentials to be stored in MineMeld? Then the only workaround I can think of is outsourcing them to an external API GW (AWS API GW in example) that could proxy the connection between MineMeld and the original feed. But you'll have just kicked your problem upstream.

Thank you very much! It worked!

  • 15071 Views
  • 20 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!