topic URLHaus complete list help in General Topics

URLHaus complete list help

Mattk — Tue, 13 Aug 2019 21:13:51 GMT

I am trying to pull the complete list from URLHaus (https://urlhaus.abuse.ch/api/) and specificly the CSV feed. (https://urlhaus.abuse.ch/downloads/csv/)

The problem is this. The feed is huge! Over 200k right now, so the PAN will not take it because of it's limits.

I noticed that most of these URL's are marked offline. So there are less than 5k of them on line. This is good.

My problem is this. How do I get a miner to only grab the online URL's? Has anyone else done this?

Re: URLHaus complete list help

lmori — Wed, 14 Aug 2019 13:15:46 GMT

Hi @Mattk,

you can do this:

- miner based on CSV class

- use ignore_regex to ignore lines with "offline" status

Example:

Prototype config:

attributes:
    confidence: 80
    share_level: green
    type: URL
fieldnames:
- urlhaus_id
- urlhaus_dateadded
- indicator
- urlhaus_url_status
- urlhaus_threat
- urlhaus_tags
- urlhaus_link
- urlhaus_reporter
ignore_regex: (?:^#)|(?:.*,"offline",)
interval: 300
source_name: urlhaus.csv
url: https://urlhaus.abuse.ch/downloads/csv/

Re: URLHaus complete list help

Mattk — Thu, 15 Aug 2019 22:09:16 GMT

You are awesome! Thank you so much!