Using Minemeld for URL EDL

[wdoria]

Dear MM comunity,

   I am trying to use MM for parsing a URL list to populate a PA NGFW which lacks Url filtering license.

I have found that predefined miner  urlhaus.URL which seems very well done. It is based on https://urlhaus.abuse.ch/ , which is free of charge.

I have cloned it, then cloned a URL aggregator and a URL Output.

I used the following aggregator

PROTOTYPE

stdlib.aggregatorURL

and the following URL output

PROTOTYPE

stdlib.feedHCWithValue

So, I obtained an output, but seems it is not useful for NGFW (running 8.1 version) , probably because of http:// in front of every URL

that is the output  (BE CAREFUL DON'T CLICK THEM)

[...]

http://0-day.us/img/exe/7.exe
http://0-day.us/img/exe/8.ex
http://0-day.us/img/puttsy.vbs 
http://00294949493yur93.space/1ishuwuycywgeacqylyik.exe
http://01.azrj-phone.zuliyego.cn/wenbenchakanqi_yxdown.com.apk

[...] 

I think I need to strip the http:// in order to be used by Panos..

For reference the queue reference the complete output is that:

https://wdoria-rg1-mm.westeurope.cloudapp.azure.com/feeds/ABUSE-feedHCWithValue

Any tips is appreciated.

Walter Doria

[xhoms · Accepted solution]

Hi @wdoria,

just add the "?v=panosurl" at the end of the output node url to get all these anonying prefixes being removed by MineMeld.

More details in https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

[dhenke]

I would like to use the urlhaus list as well, but it currently has over 90,000 entries, while the PA-5000 and PA-7000 support a maximum of 50,000 URLs.  Is there a smarter way to trim this list other than just blindly dropping the oldest entries using the "?n=50000" parameter?

[xhoms]

Hi @dhenke,

is there any "confidence-like" value attached to the indicators you could use as a input filter criteria?

[dhenke]

Unfortunately, no.

The predefined miner urlhaus.yml has a url of https://urlhaus.abuse.ch/downloads/text/, which is just a listing of malware URLs with no other values.  There is a different url at https://urlhaus.abuse.ch/downloads/csv/ that has several fields (ID, Dateadded, URL, URL status, Threat, Associated tags, and Link to URLhaus entry), but none with a confidence value.

I suppose one could re-write the miner to use the other URL and generate their own level of confidence from the "Dateadded" and "URL status" (excluding the oldest entries that have an "offline" status), but that's a little beyond my current level of proficiency.