Using Minemeld for URL EDL

L0 Member

Using Minemeld for URL EDL

Dear MM comunity,

   I am trying to use MM for parsing a URL list to populate a PA NGFW which lacks Url filtering license.

I have found that predefined miner  urlhaus.URL which seems very well done. It is based on https://urlhaus.abuse.ch/ , which is free of charge.

 

I have cloned it, then cloned a URL aggregator and a URL Output.

I used the following aggregator

PROTOTYPE stdlib.aggregatorURL

and the following URL output

PROTOTYPE stdlib.feedHCWithValue

 

So, I obtained an output, but seems it is not useful for NGFW (running 8.1 version) , probably because of http:// in front of every URL

that is the output  (BE CAREFUL DON'T CLICK THEM)

[...]

http://0-day.us/img/exe/7.exe
http://0-day.us/img/exe/8.ex
http://0-day.us/img/puttsy.vbs
http://00294949493yur93.space/1ishuwuycywgeacqylyik.exe
http://01.azrj-phone.zuliyego.cn/wenbenchakanqi_yxdown.com.apk

[...] 

 

I think I need to strip the http:// in order to be used by Panos..

 

For reference the queue reference the complete output is that:

https://wdoria-rg1-mm.westeurope.cloudapp.azure.com/feeds/ABUSE-feedHCWithValue

 

Any tips is appreciated.

Walter Doria

L5 Sessionator

Re: Using Minemeld for URL EDL

Hi @wdoria,

 

just add the "?v=panosurl" at the end of the output node url to get all these anonying prefixes being removed by MineMeld.

 

More details in https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

L1 Bithead

Re: Using Minemeld for URL EDL

I would like to use the urlhaus list as well, but it currently has over 90,000 entries, while the PA-5000 and PA-7000 support a maximum of 50,000 URLs.  Is there a smarter way to trim this list other than just blindly dropping the oldest entries using the "?n=50000" parameter?

L5 Sessionator

Re: Using Minemeld for URL EDL

Hi @dhenke,

 

is there any "confidence-like" value attached to the indicators you could use as a input filter criteria?

L1 Bithead

Re: Using Minemeld for URL EDL

Unfortunately, no.

 

The predefined miner urlhaus.yml has a url of https://urlhaus.abuse.ch/downloads/text/, which is just a listing of malware URLs with no other values.  There is a different url at https://urlhaus.abuse.ch/downloads/csv/ that has several fields (ID, Dateadded, URL, URL status, Threat, Associated tags, and Link to URLhaus entry), but none with a confidence value.

 

I suppose one could re-write the miner to use the other URL and generate their own level of confidence from the "Dateadded" and "URL status" (excluding the oldest entries that have an "offline" status), but that's a little beyond my current level of proficiency.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!