URLHaus complete list help

[Mattk]

I am trying to pull the complete list from URLHaus (https://urlhaus.abuse.ch/api/) and specificly the CSV feed. (https://urlhaus.abuse.ch/downloads/csv/)

The problem is this. The feed is huge! Over 200k right now, so the PAN will not take it because of it's limits. 

I noticed that most of these URL's are marked offline. So there are less than 5k of them on line. This is good.

My problem is this. How do I get a miner to only grab the online URL's? Has anyone else done this?

[lmori · Accepted solution]

Hi @Mattk,

you can do this:

- miner based on CSV class

- use ignore_regex to ignore lines with "offline" status

Example:

Prototype config:

attributes:
    confidence: 80
    share_level: green
    type: URL
fieldnames:
- urlhaus_id
- urlhaus_dateadded
- indicator
- urlhaus_url_status
- urlhaus_threat
- urlhaus_tags
- urlhaus_link
- urlhaus_reporter
ignore_regex: (?:^#)|(?:.*,"offline",)
interval: 300
source_name: urlhaus.csv
url: https://urlhaus.abuse.ch/downloads/csv/

[Mattk]

You are awesome! Thank you so much!