topic Custom Prototype for 'basic' weblist mining in General Topics

Custom Prototype for 'basic' weblist mining

apackard — Wed, 20 Mar 2019 20:47:48 GMT

Hi,

I'm trying to create a new custom prototype to ingest a list of URL's that is hosted on a simple webserver.

badman.one/1/2/3
badman.two/
badman.three/2/3/4/5.exe

The list is published as above, and it works perfectly when I use as an EDL source, but I cannot ingest into MineMeld (as I want to mux it with my other lists to reduce overlap etc).

I'm not getting any errors, I just get no indicators being pulled in.

The custom mining prototype config is as below. Note that I initially didn't have the indicator section as I was wondering if MM needed the 'http://' at the beginning (I'm not sure I've defined that right either)

age_out:
   default: null
   interval: 67
   sudden_death: true
attributes:
   confidence: 100
   share_level: green
indicator:
   regex: ^(.*)
   transform: http://\1
interval: 300
   source_name: feedx_urls
   url: https://webserver.acme.com/badmanlist/url.txt
   verify_cert: false

Many thanks

Re: Custom Prototype for 'basic' weblist mining

apackard — Wed, 20 Mar 2019 21:32:13 GMT

Note I have stripped out any of my 'try this' changes and have this now, including dropping back to HTTP. I have also checked I can reach the webserver from MM by performing a wget for the target URL and it successfully downloads the list.

Also note that the prototype is based on the minemeld.ft.local.YamlURLFT class.

age_out:
   default: null
   interval: 67
   sudden_death: true
attributes:
   confidence: 100
   share_level: green
   type: URL
interval: 300
   source_name: feedx.urls
   url: http://webserver.acme.com/badmanlist/url.txt

Re: Custom Prototype for 'basic' weblist mining

apackard — Wed, 20 Mar 2019 21:42:22 GMT

Ahh, idiot, as I wrote the last I realised I'd used the wrong prototype type - minemeld.ft.local.YamlURLFT not minemeld.ft.http.HttpFT.

Ignore me!