Adding filter "?v=panosurl" broken access to all websites

Reply
Highlighted
L3 Networker

Adding filter "?v=panosurl" broken access to all websites

Custom URL category is configured to block phishing URLs collected from Linux MineMeld server through EDL. For some reason adding filter "?v=panosurl" (https://10.9.0.60/feeds/phishing-url?v=panosurl) to retrieve URLs in PAN-OS supported format (malware.com) is creating issue as all the websites are categorized as phishing and blocked. Using without filter ( https://10.9.0.60/feeds/phishing-url) don't work because URLs are retrieved in format (http://malware.com)

 

Found this live community post for similar issue https://live.paloaltonetworks.com/t5/General-Topics/Adding-v-panosurl-to-MineMeld-EDL-brought-down-o...

 

What is the solution for this ?

Tags (1)
L7 Applicator

Re: Adding filter "?v=panosurl" broken access to all websites

Hi @dannadurai,

could you share which Miners are you using?

 

Luigi

L3 Networker

Re: Adding filter "?v=panosurl" broken access to all websites

Hello,

 

Using openphish miner

 

https://openphish.com/feed.txt

L3 Networker

Re: Adding filter "?v=panosurl" broken access to all websites

Thanks for the post; although I found it after I had experienced the same thing; however, my list did not include a *.com or *.it.

 

name@fw(active)>  request system external-list show type ip name edl-phishing-sites 
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

 

craigpdce

 

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

L2 Linker

Re: Adding filter "?v=panosurl" broken access to all websites

I was able to reproduce this at will on several systems running PANOS 8.1.4.

 

name@fw(active)>  request system external-list show type ip name edl-phishing-sites 
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

 

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

 

Attached the output of the "request system external-list" command. Sorry came out really huge when I converted to pdf.

L7 Applicator

Re: Adding filter "?v=panosurl" broken access to all websites

Hi @craigomatic,

thanks, that was my same result (i.e. no URLs like *.<tld>). Let me look into this.

L7 Applicator

Re: Adding filter "?v=panosurl" broken access to all websites

Hi @craigomatic,

I need some help in reproducing this issue, could you:

  1. tell me about the policy you are using to enforce the phishing list? Are you using the EDL in URL profile or directly in the security policy under URL Category? Which application and service?
  2. Could you share the dump of the feed instead of the pdf? Basically on the browser click on the URL and copy & paste the content in a plain text file
  3. Are you using openphish or other feeds?
L2 Linker

Re: Adding filter "?v=panosurl" broken access to all websites

@lmori, thanks for your response.

 

I'm using the EDL URL category in the "category" field as follows:

 

set device-group LOCATIONS pre-rulebase security rules openphish-alert target negate no
set device-group LOCATIONS pre-rulebase security rules openphish-alert to untrust
set device-group LOCATIONS pre-rulebase security rules openphish-alert from [ trust ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert source any
set device-group LOCATIONS pre-rulebase security rules openphish-alert destination any
set device-group LOCATIONS pre-rulebase security rules openphish-alert source-user any
set device-group LOCATIONS pre-rulebase security rules openphish-alert category edl-phishing-sites
set device-group LOCATIONS pre-rulebase security rules openphish-alert application [ ssl web-browsing ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert service application-default
set device-group LOCATIONS pre-rulebase security rules openphish-alert hip-profiles any
set device-group LOCATIONS pre-rulebase security rules openphish-alert action drop
set device-group LOCATIONS pre-rulebase security rules openphish-alert log-setting lf-pnrm-logr-alienv
set device-group LOCATIONS pre-rulebase security rules openphish-alert disabled yes
set device-group LOCATIONS pre-rulebase security rules openphish-alert tag

 

This particular feed is using the openphish URL feed.

 

I can't attach a .txt file so that's why it was a pdf. Here is the dump of the edl WITHOUT the ?=panosurl flag.

 

BTW before I do this, do you really want me to paste >2000 lines into this thread? Here's a sneak preview:

 

vsys1/edl-phishing-sites:
Next update at : Fri Jan 4 09:00:36 2019
Source : https://10.X.X.X/feeds/phishing-url
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2108
Total invalid entries : 74
Valid urls:
http://venuesearch.in/include/santan/details.htm
http://venuesearch.in/include/santan/questions.htm
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/final.php
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/num.php
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://30dayaffiliatechallenge.com/wp-content/plugins/revslider/admin/views/system/.../china/?login=...
http://cesy.edu.mx/wp-admin/css/colors/chase/chase/home/auth/index.php
http://city-sm.ru/administrator/components/com_installer/helpers/mtch/match
http://d2gconsult.com.br/1c543ee7f3c60f6a7047d71b999da39dMzQzZjEzYjRmNmQ2MzNmYmJkZDMyNjc2NGNlMjRjMTM...
http://imroadrunner.com/dev/www.loginalibaba.com/alibaba/alibaba/login.alibaba.com.php?email=anko@an...
http://ledutech.org.br/logs/js/fix/f8870
http://msrebeco.cl/A1/dl/DHL.htm
http://nygift19.com/atb/index.html
http://nygift19.com/atb/questions.html
http://nygift19.com/bnc/National%20Bank%20Online.html
http://nygift19.com/cibc/index.php
http://nygift19.com/cibc/question.html
http://seediest-aids.000webhostapp.com/2/authws/1/login.php
http://venuesearch.in/include/santan/ssanta.htm

 

 

L7 Applicator

Re: Adding filter "?v=panosurl" broken access to all websites

Hi @craigomatic,

would you mind sending me the .txt file over at lmori@paloaltonetworks.com?

I am trying to reproduce the issue, checking if it is an issue on MM or in PAN-OS.

 

Luigi

L2 Linker

Re: Adding filter "?v=panosurl" broken access to all websites

Sure it's on the way. Also included the MineMeld config.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!