Custom Prototype for 'basic' weblist mining

L4 Transporter

Custom Prototype for 'basic' weblist mining

Hi,

 

I'm trying to create a new custom prototype to ingest a list of URL's that is hosted on a simple webserver.

 

badman.one/1/2/3
badman.two/
badman.three/2/3/4/5.exe

The list is published as above, and it works perfectly when I use as an EDL source, but I cannot ingest into MineMeld (as I want to mux it with my other lists to reduce overlap etc).

 

I'm not getting any errors, I just get no indicators being pulled in.

 

The custom mining prototype config is as below.  Note that I initially didn't have the indicator section as I was wondering if MM needed the 'http://' at the beginning (I'm not sure I've defined that right either)

 

age_out:
   default: null
   interval: 67
   sudden_death: true
attributes:
   confidence: 100
   share_level: green
indicator:
   regex: ^(.*)
   transform: http://\1
interval: 300
   source_name: feedx_urls
   url: https://webserver.acme.com/badmanlist/url.txt
   verify_cert: false  

 

Many thanks

L4 Transporter

Re: Custom Prototype for 'basic' weblist mining

Note I have stripped out any of my 'try this' changes and have this now, including dropping back to HTTP.  I have also checked I can reach the webserver from MM by performing a wget for the target URL and it successfully downloads the list.

 

Also note that the prototype is based on the minemeld.ft.local.YamlURLFT class.

 

age_out:
   default: null
   interval: 67
   sudden_death: true
attributes:
   confidence: 100
   share_level: green
   type: URL
interval: 300
   source_name: feedx.urls
   url: http://webserver.acme.com/badmanlist/url.txt

L4 Transporter

Re: Custom Prototype for 'basic' weblist mining

Ahh, idiot, as I wrote the last I realised I'd used the wrong prototype type - minemeld.ft.local.YamlURLFT not minemeld.ft.http.HttpFT.

 

Ignore me!

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!