MineMeld Discussions

Reply
L1 Bithead
Posts: 11
Registered: ‎09-28-2016
Accepted Solution

JSON Miner with basic auth and an API key

I would like to have a miner to connect to the Pingdom API to pull a list of their US node addresses.   The Pingdom API uses basic auth and requires an "app key".

 

Their documentation references two HTTP headers, authentication and app-key:

> GET /checks HTTP/1.1
> Host: api.pingdom.com
> Authentication: Zm9vQGV4YW1wbGUuY29tOnBhc3N3b3Jk
> App-Key: 1234567890abcdef1234567890abcdef

 

Using a JSON miner, encoded our login (username:password) with Base64 encoding (my assumption on what they were expecting based on reading their info) and generated an app key via their portal.

 

However, I am not sure how I add these attributes to the miner.

 

Thank you.

L1 Bithead
Posts: 11
Registered: ‎09-28-2016

Re: JSON Miner with basic auth and an API key

This is the configuration I currently have for the miner, which I also need to restrict to US based addresses:

 

pingdom_nodes-US:
class: minemeld.ft.json.SimpleJSON
config:
age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: probes
fields:
- id
- country
- city
- name
- active
- hostname
- ip
- countryiso
headers:
- authorization Basic <base64 encoded string>
- app-key <app key>
indicator: ip
outfilters:
- actions:
- accept
conditions:
- countryiso == 'US'
name: accept US
- actions:
- drop
name: drop all
url: https://api.pingdom.com/api/2.1/probes
description: Pingdom nodes
development_status: EXPERIMENTAL
indicator_types:
- IPv4
node_type: miner
tags:
- ConfidenceHigh
- ShareLevelGreen

L1 Bithead
Posts: 11
Registered: ‎09-28-2016

Re: JSON Miner with basic auth and an API key

I've verified the app-key and authorization string using curl, but I am getting an error with the node.  The error is 

'list' object has no attribute 'items'
L1 Bithead
Posts: 11
Registered: ‎09-28-2016

Re: JSON Miner with basic auth and an API key

Fixed the list issue and now the miner is showing 111 indicators

 

The final configuration of the miner is

    pingdom_nodes-US:
        class: minemeld.ft.json.SimpleJSON
        config:
            age_out:
                default: null
                interval: 257
                sudden_death: true
            attributes:
                confidence: 100
                share_level: green
                type: IPv4
            interval: 3600
            extractor: probes
            fields:
            - id
            - country
            - city
            - name
            - active
            - hostname
            - ip
            - countryiso
            headers:
              Authorization: Basic <base64>
              App-Key: <app-key>
            indicator: ip
            outfilters:
            -   actions:
                - accept
                conditions:
                - countryiso == 'US'
                name: accept US
            -   actions:
                - drop
                name: drop all
            url: https://api.pingdom.com/api/2.1/probes
        description: Pingdom nodes
        development_status: STABLE
        indicator_types:
        - IPv4
        node_type: miner
        tags:
        - ConfidenceHigh
        - ShareLevelGreen

The problem now is that neither the aggregator nor the feed have any entries.

 

What is confusing is that if I pull the data via curl, I see output like this:

{"probes":[{"id":36,"country":"United States","city":"Chicago","name":"Chicago, IL","active":true,"hostname":"s414.pingdom.com","ip":"174.34.156.130","countryiso":"US","region":"NA"},
{"id":43,"country":"United States","city":"Denver","name":"Denver, CO","active":true,"hostname":"s421.pingdom.com","ip":"173.248.147.18","countryiso":"US","ipv6":"2607:fc88:100:26::421","region":"NA"},
{"id":47,"country":"Austria","city":"Vienna","name":"Vienna, Austria","active":true,"hostname":"s425.pingdom.com","ip":"178.255.152.2","countryiso":"AT","ipv6":"2a00:11c0:d:3::425","region":"EU"},

In the logs for the miner though, I see

 

{
    "_age_out": 4294967295000,
    "confidence": 100,
    "json_hostname": "s4054.pingdom.com",
    "json_country": "South Korea",
    "share_level": "green",
    "json_ip": "169.56.174.147",
    "_last_run": 1548903839226,
    "json_city": "Seoul",
    "sources": [
        "pingdom_nodes-US"
    ],
    "json_name": "Seoul",
    "json_active": false,
    "json_id": 259,
    "first_seen": 1548903839226,
    "json_countryiso": "KR",
    "type": "IPv4",
    "json_region": "APAC",
    "last_seen": 1548903839226
}

If I change the field names in the miner to "json_*", the miner shows 0 indicators.

 

My aggregator looks like this

aggregator.jpg

 

The feed looks like this

feed.jpgThank you in advance.

 

Mike

Highlighted
L1 Bithead
Posts: 11
Registered: ‎09-28-2016

Re: JSON Miner with basic auth and an API key

I went back to the article Using MineMeld to extract indicators from a generic API and re-read the JSON section.  It noted the addition of the json_ prefix to the field names.

 

Changing my match criteria to use the prefix resolved the issue.

 

It is working as expected.

L7 Applicator
Posts: 1,036
Registered: ‎03-03-2011

Re: JSON Miner with basic auth and an API key

Hi @MichaelPDean,

would you consider a Pull Request to our github repo for prototypes?

https://github.com/PaloAltoNetworks/minemeld-node-prototypes

L1 Bithead
Posts: 11
Registered: ‎09-28-2016

Re: JSON Miner with basic auth and an API key

Would this potentially change anything I created that is based on a system/standard prototype?

 

I am open to it either way, but need to know if I have to watch out for potential changes to custom nodes.

 

What is the command to do the pull and have the updates completed?

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!