MineMeld Speedtest.net host mining

[oschuler]

Hi all,

I managed to install MineMeld on-prem and are playing around with it now. As a first task I'd like to setup a domain feed delivering SpeedTest.net hosts from countries we deployed Palo Alto firewalls.

Setting up the miner, a domain aggregator and an output worked, no problem there. We download the full hosts list from http://c.speedtest.net/speedtest-servers-static.php and pass it through to an output feed. However, I don't need all the 7k+ hosts mined. Only those with specific two-digit country codes, such as "CH" or "US". That information is also present in the mined XML file and I tried to fetch this information using "fields" in the input config (field 'countrycode'):

age_out:
  default: null
  interval: 631
  sudden_death: true
attributes:
  confidence: 100
  direction: inbound
  share_level: green
  type: domain
fields:
  country: 

    regex: country="([\w\s]*)"
    transform: \1
  countrycode:
    regex: cc="(\w){2}"
    transform: \1
ignore_regex: ^\<settings\>|^\<servers\>
indicator:
  regex: host="(.*):8080"
  transform: \1
interval: 3307
source_name: speedtest.hosts
url: http://c.speedtest.net/speedtest-servers-static.php

Now I want to filter for this 'countrycode' field in the output node and created a new prototype for this (condition #2):

infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- countrycode == 'CH'
- share_level == 'green'
name: accept share level green
- actions:
- drop
name: drop all

Result: 0 indicators in that output feed. Thanks for any hints on this. I'd like to avoid creating a miner for every country.

Thanks,

Oliver

[oschuler · Accepted solution]

Implemented the following solution now. Filtering all the countries we need directly in the input node. However, it could be desirable to filter the output instead of input.

age_out:
default: null
interval: last_seen+900
sudden_death: true
attributes:
confidence: 100
direction: inbound
share_level: green
type: domain
ignore_regex: ^\<settings\>|^\<servers\>
indicator:
regex: cc="(CH|US|<some more countries>)".*host="(.*):8080"
transform: \2
interval: 86400
source_name: speedtest.hosts
url: http://c.speedtest.net/speedtest-servers-static.php