Using Minemeld for URL EDL

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Using Minemeld for URL EDL

L0 Member

Dear MM comunity,

   I am trying to use MM for parsing a URL list to populate a PA NGFW which lacks Url filtering license.

I have found that predefined miner  urlhaus.URL which seems very well done. It is based on https://urlhaus.abuse.ch/ , which is free of charge.

 

I have cloned it, then cloned a URL aggregator and a URL Output.

I used the following aggregator

PROTOTYPE stdlib.aggregatorURL

and the following URL output

PROTOTYPE stdlib.feedHCWithValue

 

So, I obtained an output, but seems it is not useful for NGFW (running 8.1 version) , probably because of http:// in front of every URL

that is the output  (BE CAREFUL DON'T CLICK THEM)

[...]

http://0-day.us/img/exe/7.exe
http://0-day.us/img/exe/8.ex
http://0-day.us/img/puttsy.vbs
http://00294949493yur93.space/1ishuwuycywgeacqylyik.exe
http://01.azrj-phone.zuliyego.cn/wenbenchakanqi_yxdown.com.apk

[...] 

 

I think I need to strip the http:// in order to be used by Panos..

 

For reference the queue reference the complete output is that:

https://wdoria-rg1-mm.westeurope.cloudapp.azure.com/feeds/ABUSE-feedHCWithValue

 

Any tips is appreciated.

Walter Doria

1 accepted solution

Accepted Solutions

L5 Sessionator

Hi @wdoria,

 

just add the "?v=panosurl" at the end of the output node url to get all these anonying prefixes being removed by MineMeld.

 

More details in https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

View solution in original post

4 REPLIES 4

L5 Sessionator

Hi @wdoria,

 

just add the "?v=panosurl" at the end of the output node url to get all these anonying prefixes being removed by MineMeld.

 

More details in https://live.paloaltonetworks.com/t5/MineMeld-Articles/Parameters-for-the-output-feeds/ta-p/146170

I would like to use the urlhaus list as well, but it currently has over 90,000 entries, while the PA-5000 and PA-7000 support a maximum of 50,000 URLs.  Is there a smarter way to trim this list other than just blindly dropping the oldest entries using the "?n=50000" parameter?

Hi @dhenke,

 

is there any "confidence-like" value attached to the indicators you could use as a input filter criteria?

Unfortunately, no.

 

The predefined miner urlhaus.yml has a url of https://urlhaus.abuse.ch/downloads/text/, which is just a listing of malware URLs with no other values.  There is a different url at https://urlhaus.abuse.ch/downloads/csv/ that has several fields (ID, Dateadded, URL, URL status, Threat, Associated tags, and Link to URLhaus entry), but none with a confidence value.

 

I suppose one could re-write the miner to use the other URL and generate their own level of confidence from the "Dateadded" and "URL status" (excluding the oldest entries that have an "offline" status), but that's a little beyond my current level of proficiency.

  • 1 accepted solution
  • 10206 Views
  • 4 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!