Miner for host file format.

[DatacomNetadmin]

Is there a miner + documentation on how to get it working for a host file list?

i.e.

https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

This tool was recommended by Palo Alto for a project we are working on, howver the documentation on how to actually use it is hard to understand.

[xhoms]

Hi @DatacomNetadmin,

you can use the generic HttpFT class miner for such a lists published through HTTP/S. The following are the steps to mine the list at https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

A- Create a new prototype for StevenBlack's list

1. Locate in the configuration any prototype using the HttpFT class. For instance the "auscert.1day_dumpsites" one.
2. Click on "new" to create a new prototype and name it "StevenBlack" (or anything else that suits you)
   
3. Replace the configuration of the new prototype with this one:
   

   age_out:
       default: null
       interval: 3600
       sudden_death: true
   attributes:
       confidence: 100
       direction: inbound
       interval: 3600
       share_level: green
       type: domain
   ignore_regex: ^#
   indicator:
       regex: ^0\.0\.0\.0[\s\t](.*\.[a-z]{2,})$
       transform: \1
   source_name: StevenBlack
   url: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

B- Clone the just created prototype into a working node.

Just locate the new prototype in the configuration and use the "Clone" option.

[DatacomNetadmin · Accepted solution]

Hi xhoms,

Thank you,

I can  see where I got my miner wrong.

I had the wrong indicator type (URL), I changed it to domain and changed the aggergator to suit.

I have it working now using the stdlib.aggregatorDomain aggregator and the stdlib.feedHCGreen protype for the output.

Again, much appreciated.

DaveC