MineMeld Speedtest.net host mining

oschuler — Tue, 01 May 2018 07:39:34 GMT

Hi all,

I managed to install MineMeld on-prem and are playing around with it now. As a first task I'd like to setup a domain feed delivering SpeedTest.net hosts from countries we deployed Palo Alto firewalls.

Setting up the miner, a domain aggregator and an output worked, no problem there. We download the full hosts list from http://c.speedtest.net/speedtest-servers-static.php and pass it through to an output feed. However, I don't need all the 7k+ hosts mined. Only those with specific two-digit country codes, such as "CH" or "US". That information is also present in the mined XML file and I tried to fetch this information using "fields" in the input config (field 'countrycode'):

age_out:
default: null
interval: 631
sudden_death: true
attributes:
confidence: 100
direction: inbound
share_level: green
type: domain
fields:
country:

regex: country="([\w\s]*)"
transform: \1
countrycode:
regex: cc="(\w){2}"
transform: \1
ignore_regex: ^\<settings\>|^\<servers\>
indicator:
regex: host="(.*):8080"
transform: \1
interval: 3307
source_name: speedtest.hosts
url: http://c.speedtest.net/speedtest-servers-static.php

Now I want to filter for this 'countrycode' field in the output node and created a new prototype for this (condition #2):

infilters:
- actions:
- accept
conditions:
- __method == 'withdraw'
name: accept withdraws
- actions:
- accept
conditions:
- countrycode == 'CH'
- share_level == 'green'
name: accept share level green
- actions:
- drop
name: drop all

Result: 0 indicators in that output feed. Thanks for any hints on this. I'd like to avoid creating a miner for every country.

Thanks,

Oliver

Re: MineMeld Speedtest.net host mining

oschuler — Thu, 03 May 2018 14:26:26 GMT

Implemented the following solution now. Filtering all the countries we need directly in the input node. However, it could be desirable to filter the output instead of input.

age_out:
default: null
interval: last_seen+900
sudden_death: true
attributes:
confidence: 100
direction: inbound
share_level: green
type: domain
ignore_regex: ^\<settings\>|^\<servers\>
indicator:
regex: cc="(CH|US|<some more countries>)".*host="(.*):8080"
transform: \2
interval: 86400
source_name: speedtest.hosts
url: http://c.speedtest.net/speedtest-servers-static.php

topic MineMeld Speedtest.net host mining in General Topics

MineMeld Speedtest.net host mining

Re: MineMeld Speedtest.net host mining