topic Re: JSON Miner with basic auth and an API key in General Topics

JSON Miner with basic auth and an API key

MichaelPDean — Wed, 30 Jan 2019 01:20:11 GMT

I would like to have a miner to connect to the Pingdom API to pull a list of their US node addresses. The Pingdom API uses basic auth and requires an "app key".

Their documentation references two HTTP headers, authentication and app-key:

> GET /checks HTTP/1.1
> Host: api.pingdom.com
> Authentication: Zm9vQGV4YW1wbGUuY29tOnBhc3N3b3Jk
> App-Key: 1234567890abcdef1234567890abcdef

Using a JSON miner, encoded our login (username:password) with Base64 encoding (my assumption on what they were expecting based on reading their info) and generated an app key via their portal.

However, I am not sure how I add these attributes to the miner.

Thank you.

Re: JSON Miner with basic auth and an API key

MichaelPDean — Wed, 30 Jan 2019 01:22:59 GMT

This is the configuration I currently have for the miner, which I also need to restrict to US based addresses:

pingdom_nodes-US:
class: minemeld.ft.json.SimpleJSON
config:
age_out:
default: null
interval: 257
sudden_death: true
attributes:
confidence: 100
share_level: green
type: IPv4
extractor: probes
fields:
- id
- country
- city
- name
- active
- hostname
- ip
- countryiso
headers:
- authorization Basic <base64 encoded string>
- app-key <app key>
indicator: ip
outfilters:
- actions:
- accept
conditions:
- countryiso == 'US'
name: accept US
- actions:
- drop
name: drop all
url: https://api.pingdom.com/api/2.1/probes
description: Pingdom nodes
development_status: EXPERIMENTAL
indicator_types:
- IPv4
node_type: miner
tags:
- ConfidenceHigh
- ShareLevelGreen

Re: JSON Miner with basic auth and an API key

MichaelPDean — Thu, 31 Jan 2019 01:16:22 GMT

I've verified the app-key and authorization string using curl, but I am getting an error with the node. The error is

'list' object has no attribute 'items'

Re: JSON Miner with basic auth and an API key

MichaelPDean — Thu, 31 Jan 2019 03:09:58 GMT

Fixed the list issue and now the miner is showing 111 indicators

The final configuration of the miner is

    pingdom_nodes-US:
        class: minemeld.ft.json.SimpleJSON
        config:
            age_out:
                default: null
                interval: 257
                sudden_death: true
            attributes:
                confidence: 100
                share_level: green
                type: IPv4
            interval: 3600
            extractor: probes
            fields:
            - id
            - country
            - city
            - name
            - active
            - hostname
            - ip
            - countryiso
            headers:
              Authorization: Basic <base64>
              App-Key: <app-key>
            indicator: ip
            outfilters:
            -   actions:
                - accept
                conditions:
                - countryiso == 'US'
                name: accept US
            -   actions:
                - drop
                name: drop all
            url: https://api.pingdom.com/api/2.1/probes
        description: Pingdom nodes
        development_status: STABLE
        indicator_types:
        - IPv4
        node_type: miner
        tags:
        - ConfidenceHigh
        - ShareLevelGreen

The problem now is that neither the aggregator nor the feed have any entries.

What is confusing is that if I pull the data via curl, I see output like this:

{"probes":[{"id":36,"country":"United States","city":"Chicago","name":"Chicago, IL","active":true,"hostname":"s414.pingdom.com","ip":"174.34.156.130","countryiso":"US","region":"NA"},
{"id":43,"country":"United States","city":"Denver","name":"Denver, CO","active":true,"hostname":"s421.pingdom.com","ip":"173.248.147.18","countryiso":"US","ipv6":"2607:fc88:100:26::421","region":"NA"},
{"id":47,"country":"Austria","city":"Vienna","name":"Vienna, Austria","active":true,"hostname":"s425.pingdom.com","ip":"178.255.152.2","countryiso":"AT","ipv6":"2a00:11c0:d:3::425","region":"EU"},

In the logs for the miner though, I see

{
    "_age_out": 4294967295000,
    "confidence": 100,
    "json_hostname": "s4054.pingdom.com",
    "json_country": "South Korea",
    "share_level": "green",
    "json_ip": "169.56.174.147",
    "_last_run": 1548903839226,
    "json_city": "Seoul",
    "sources": [
        "pingdom_nodes-US"
    ],
    "json_name": "Seoul",
    "json_active": false,
    "json_id": 259,
    "first_seen": 1548903839226,
    "json_countryiso": "KR",
    "type": "IPv4",
    "json_region": "APAC",
    "last_seen": 1548903839226
}

If I change the field names in the miner to "json_*", the miner shows 0 indicators.

My aggregator looks like this

The feed looks like this

Thank you in advance.

Mike

Re: JSON Miner with basic auth and an API key

MichaelPDean — Thu, 31 Jan 2019 20:09:51 GMT

I went back to the article Using MineMeld to extract indicators from a generic API and re-read the JSON section. It noted the addition of the json_ prefix to the field names.

Changing my match criteria to use the prefix resolved the issue.

It is working as expected.

Re: JSON Miner with basic auth and an API key

lmori — Thu, 07 Feb 2019 08:11:02 GMT

Hi @MichaelPDean,

would you consider a Pull Request to our github repo for prototypes?

https://github.com/PaloAltoNetworks/minemeld-node-prototypes

Re: JSON Miner with basic auth and an API key

MichaelPDean — Thu, 07 Feb 2019 19:29:19 GMT

Would this potentially change anything I created that is based on a system/standard prototype?

I am open to it either way, but need to know if I have to watch out for potential changes to custom nodes.

What is the command to do the pull and have the updates completed?

Re: JSON Miner with basic auth and an API key

jonnynux — Tue, 14 Dec 2021 11:06:07 GMT

Hi,

I'm adding to this discussion to describe a similar problem. I'm building a custom miner to a service that requires basic authentication. The resource contains a simple flat file with a indicator in every line.

With curl I execute the request like this:

curl -k -H 'Authorization: Basic <base64>' https://host/siem/url/apt

And the miner configuration is this:

URLS_TIP3:
    class: minemeld.ft.http.HttpFT
    config:
        age_out:
            default: null
            interval: 600
            sudden_death: true
        attributes:
            confidence: 80
            direction: inbound
            share_level: green
            type: URL
        headers:
            Authorization: Basic <base64>
        indicator:
            regex: ^http.*
        source_name: tip.URLS
        url: https://host/siem/url/apt
        verify_cert: false
    description: Tip siem url apt
    development_status: STABLE
    indicator_types:
    - URL
    node_type: miner
    tags:
    - ConfidenceHigh
    - ShareLevelGreen

But I receive in LAST RUN miner row:

401: Client Error: Unouthorized for url: https://host/siem/url/apt

Am I missing something?

Thank you.

Re: JSON Miner with basic auth and an API key

jonnynux — Tue, 14 Dec 2021 12:13:14 GMT

I answer in myself, I replaced

headers:
    Authorization: Basic <base64>

with:

username: usernamepassword: password

and everything worked perfectly.