topic Miner for host file format. in General Topics

Miner for host file format.

DatacomNetadmin — Mon, 29 Jan 2018 00:24:02 GMT

Is there a miner + documentation on how to get it working for a host file list?

i.e.

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

This tool was recommended by Palo Alto for a project we are working on, howver the documentation on how to actually use it is hard to understand.

Re: Miner for host file format.

xhoms — Mon, 29 Jan 2018 07:50:16 GMT

Hi @DatacomNetadmin,

you can use the generic HttpFT class miner for such a lists published through HTTP/S. The following are the steps to mine the list at https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

A- Create a new prototype for StevenBlack's list

Locate in the configuration any prototype using the HttpFT class. For instance the "auscert.1day_dumpsites" one.
Click on "new" to create a new prototype and name it "StevenBlack" (or anything else that suits you)

Replace the configuration of the new prototype with this one:

age_out:
    default: null
    interval: 3600
    sudden_death: true
attributes:
    confidence: 100
    direction: inbound
    interval: 3600
    share_level: green
    type: domain
ignore_regex: ^#
indicator:
    regex: ^0\.0\.0\.0[\s\t](.*\.[a-z]{2,})$
    transform: \1
source_name: StevenBlack
url: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

B- Clone the just created prototype into a working node.

Just locate the new prototype in the configuration and use the "Clone" option.

Re: Miner for host file format.

DatacomNetadmin — Tue, 30 Jan 2018 02:07:06 GMT

Hi xhoms,

Thank you,

I can see where I got my miner wrong.

I had the wrong indicator type (URL), I changed it to domain and changed the aggergator to suit.

I have it working now using the stdlib.aggregatorDomain aggregator and the stdlib.feedHCGreen protype for the output.

Again, much appreciated.

DaveC