topic Re: Google Safebrowsing miner in General Topics

Google Safebrowsing miner

chirss — Tue, 03 Oct 2017 18:58:04 GMT

Google has a threat list api, has anyone created a miner for it?

https://developers.google.com/safe-browsing/v4/lists

Re: Google Safebrowsing miner

xhoms — Thu, 05 Oct 2017 13:48:37 GMT

@chirss Google Safe Browsing lists are not really "lists". It is an API that will give you information about a given URL. I mean: you have a URL and you're wondering what Google's Safe Browsing thinks about that URL. You can use the API for such a case.

I'm planning an "Enrichement Framework" for MineMeld that will be able to attach additional attributes to indicators. A Google Safe Browsing node for the Enrichement Framework would be awesome.

Re: Google Safebrowsing miner

chirss — Thu, 05 Oct 2017 15:31:11 GMT

Ya that's what I want as well. If I can compare url information from a feed with what safebrowsing thinks of it and then come up with a ranking to be used by different outputs that would be ideal. Is this what you are thinking? I haven't played enough with miner creation to build anything like this out.

Re: Google Safebrowsing miner

chirss — Thu, 05 Oct 2017 15:32:51 GMT

Also maybe a miner isn't the right thing so much as a processor. If an ioc hits the processor it then queries the api (within limits of the api).

There are an awful lot of reputation type things which could possibly be used in a similar manner.

Re: Google Safebrowsing miner

xhoms — Thu, 05 Oct 2017 15:44:16 GMT

You're following my same path.

I started thinking on miners calling enrichement API's to attach additional attributes to the indicators.
Then I though it would be better implemented as an aggregator node (once for many nodes)
Then I realized a same indicator (i.e. a URL) could be enriched by many sources (i.e. Safe Browsing, PAN-DB, etc.) and that a cache should be put in place to avoid continous calls for the same indicatos.

This is why I reached to the point that a "Enrichement Framework" for MineMeld would be welcome by the community. So I have it in my current plan of intentions.

Re: Google Safebrowsing miner

chirss — Thu, 05 Oct 2017 16:50:44 GMT

Ya exactly.

The problem I'm finding is a lot of the miners likely have duplicate entries of some kind. So I'm sending them all to the same processor for similar types of feeds (phishing type miners to phishing processor for example). However I have to validate everything coming in before being able to trust it, i.e. verify before trusting.

The scenario you're talking about would be very beneficial in at least this scenario.