JSON Miner with basic auth and an API key

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

JSON Miner with basic auth and an API key

L2 Linker

I would like to have a miner to connect to the Pingdom API to pull a list of their US node addresses.   The Pingdom API uses basic auth and requires an "app key".

 

Their documentation references two HTTP headers, authentication and app-key:

> GET /checks HTTP/1.1
> Host: api.pingdom.com
> Authentication: Zm9vQGV4YW1wbGUuY29tOnBhc3N3b3Jk
> App-Key: 1234567890abcdef1234567890abcdef

 

Using a JSON miner, encoded our login (username:password) with Base64 encoding (my assumption on what they were expecting based on reading their info) and generated an app key via their portal.

 

However, I am not sure how I add these attributes to the miner.

 

Thank you.

1 accepted solution

Accepted Solutions

I went back to the article Using MineMeld to extract indicators from a generic API and re-read the JSON section.  It noted the addition of the json_ prefix to the field names.

 

Changing my match criteria to use the prefix resolved the issue.

 

It is working as expected.

View solution in original post

8 REPLIES 8

L2 Linker

This is the configuration I currently have for the miner, which I also need to restrict to US based addresses:

 

pingdom_nodes-US:
class: minemeld.ft.json.SimpleJSON
config:
age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: probes
fields:
- id
- country
- city
- name
- active
- hostname
- ip
- countryiso
headers:
- authorization Basic <base64 encoded string>
- app-key <app key>
indicator: ip
outfilters:
- actions:
- accept
conditions:
- countryiso == 'US'
name: accept US
- actions:
- drop
name: drop all
url: https://api.pingdom.com/api/2.1/probes
description: Pingdom nodes
development_status: EXPERIMENTAL
indicator_types:
- IPv4
node_type: miner
tags:
- ConfidenceHigh
- ShareLevelGreen

I've verified the app-key and authorization string using curl, but I am getting an error with the node.  The error is 

'list' object has no attribute 'items'

Fixed the list issue and now the miner is showing 111 indicators

 

The final configuration of the miner is

    pingdom_nodes-US:
        class: minemeld.ft.json.SimpleJSON
        config:
            age_out:
                default: null
                interval: 257
                sudden_death: true
            attributes:
                confidence: 100
                share_level: green
                type: IPv4
            interval: 3600
            extractor: probes
            fields:
            - id
            - country
            - city
            - name
            - active
            - hostname
            - ip
            - countryiso
            headers:
              Authorization: Basic <base64>
              App-Key: <app-key>
            indicator: ip
            outfilters:
            -   actions:
                - accept
                conditions:
                - countryiso == 'US'
                name: accept US
            -   actions:
                - drop
                name: drop all
            url: https://api.pingdom.com/api/2.1/probes
        description: Pingdom nodes
        development_status: STABLE
        indicator_types:
        - IPv4
        node_type: miner
        tags:
        - ConfidenceHigh
        - ShareLevelGreen

The problem now is that neither the aggregator nor the feed have any entries.

 

What is confusing is that if I pull the data via curl, I see output like this:

{"probes":[{"id":36,"country":"United States","city":"Chicago","name":"Chicago, IL","active":true,"hostname":"s414.pingdom.com","ip":"174.34.156.130","countryiso":"US","region":"NA"},
{"id":43,"country":"United States","city":"Denver","name":"Denver, CO","active":true,"hostname":"s421.pingdom.com","ip":"173.248.147.18","countryiso":"US","ipv6":"2607:fc88:100:26::421","region":"NA"},
{"id":47,"country":"Austria","city":"Vienna","name":"Vienna, Austria","active":true,"hostname":"s425.pingdom.com","ip":"178.255.152.2","countryiso":"AT","ipv6":"2a00:11c0:d:3::425","region":"EU"},

In the logs for the miner though, I see

 

{
    "_age_out": 4294967295000,
    "confidence": 100,
    "json_hostname": "s4054.pingdom.com",
    "json_country": "South Korea",
    "share_level": "green",
    "json_ip": "169.56.174.147",
    "_last_run": 1548903839226,
    "json_city": "Seoul",
    "sources": [
        "pingdom_nodes-US"
    ],
    "json_name": "Seoul",
    "json_active": false,
    "json_id": 259,
    "first_seen": 1548903839226,
    "json_countryiso": "KR",
    "type": "IPv4",
    "json_region": "APAC",
    "last_seen": 1548903839226
}

If I change the field names in the miner to "json_*", the miner shows 0 indicators.

 

My aggregator looks like this

aggregator.jpg

 

The feed looks like this

feed.jpgThank you in advance.

 

Mike

I went back to the article Using MineMeld to extract indicators from a generic API and re-read the JSON section.  It noted the addition of the json_ prefix to the field names.

 

Changing my match criteria to use the prefix resolved the issue.

 

It is working as expected.

Hi @MichaelPDean,

would you consider a Pull Request to our github repo for prototypes?

https://github.com/PaloAltoNetworks/minemeld-node-prototypes

Would this potentially change anything I created that is based on a system/standard prototype?

 

I am open to it either way, but need to know if I have to watch out for potential changes to custom nodes.

 

What is the command to do the pull and have the updates completed?

L1 Bithead

Hi,

I'm adding to this discussion to describe a similar problem. I'm building a custom miner to a service that requires basic authentication. The resource contains a simple flat file with a indicator in every line.

With curl I execute the request like this:

curl -k -H 'Authorization: Basic <base64>' https://host/siem/url/apt

And the miner configuration is this:

URLS_TIP3:
class: minemeld.ft.http.HttpFT
config:
age_out:
default: null
interval: 600
sudden_death: true
attributes:
confidence: 80
direction: inbound
share_level: green
type: URL
headers:
Authorization: Basic <base64>
indicator:
regex: ^http.*
source_name: tip.URLS
url: https://host/siem/url/apt
verify_cert: false
description: Tip siem url apt
development_status: STABLE
indicator_types:
- URL
node_type: miner
tags:
- ConfidenceHigh
- ShareLevelGreen

 But I receive in LAST RUN miner row:

401: Client Error: Unouthorized for url: https://host/siem/url/apt

 Am I missing something?

 

Thank you.

L1 Bithead

I answer in myself, I replaced

headers:
Authorization: Basic <base64>

 with:

username: usernamepassword: password

and everything worked perfectly.

  • 1 accepted solution
  • 10983 Views
  • 8 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!