Adding filter "?v=panosurl" broken access to all websites

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Adding filter "?v=panosurl" broken access to all websites

L3 Networker

Custom URL category is configured to block phishing URLs collected from Linux MineMeld server through EDL. For some reason adding filter "?v=panosurl" (https://10.9.0.60/feeds/phishing-url?v=panosurl) to retrieve URLs in PAN-OS supported format (malware.com) is creating issue as all the websites are categorized as phishing and blocked. Using without filter ( https://10.9.0.60/feeds/phishing-url) don't work because URLs are retrieved in format (http://malware.com)

 

Found this live community post for similar issue https://live.paloaltonetworks.com/t5/General-Topics/Adding-v-panosurl-to-MineMeld-EDL-brought-down-o...

 

What is the solution for this ?

21 REPLIES 21

L7 Applicator

Hi @dannadurai,

could you share which Miners are you using?

 

Luigi

Hello,

 

Using openphish miner

 

https://openphish.com/feed.txt

Thanks for the post; although I found it after I had experienced the same thing; however, my list did not include a *.com or *.it.

 

name@fw(active)>  request system external-list show type ip name edl-phishing-sites 
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

 

craigpdce

 

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

I was able to reproduce this at will on several systems running PANOS 8.1.4.

 

name@fw(active)>  request system external-list show type ip name edl-phishing-sites 
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

 

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

 

Attached the output of the "request system external-list" command. Sorry came out really huge when I converted to pdf.

Hi @craigomatic,

thanks, that was my same result (i.e. no URLs like *.<tld>). Let me look into this.

Hi @craigomatic,

I need some help in reproducing this issue, could you:

  1. tell me about the policy you are using to enforce the phishing list? Are you using the EDL in URL profile or directly in the security policy under URL Category? Which application and service?
  2. Could you share the dump of the feed instead of the pdf? Basically on the browser click on the URL and copy & paste the content in a plain text file
  3. Are you using openphish or other feeds?

@lmori, thanks for your response.

 

I'm using the EDL URL category in the "category" field as follows:

 

set device-group LOCATIONS pre-rulebase security rules openphish-alert target negate no
set device-group LOCATIONS pre-rulebase security rules openphish-alert to untrust
set device-group LOCATIONS pre-rulebase security rules openphish-alert from [ trust ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert source any
set device-group LOCATIONS pre-rulebase security rules openphish-alert destination any
set device-group LOCATIONS pre-rulebase security rules openphish-alert source-user any
set device-group LOCATIONS pre-rulebase security rules openphish-alert category edl-phishing-sites
set device-group LOCATIONS pre-rulebase security rules openphish-alert application [ ssl web-browsing ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert service application-default
set device-group LOCATIONS pre-rulebase security rules openphish-alert hip-profiles any
set device-group LOCATIONS pre-rulebase security rules openphish-alert action drop
set device-group LOCATIONS pre-rulebase security rules openphish-alert log-setting lf-pnrm-logr-alienv
set device-group LOCATIONS pre-rulebase security rules openphish-alert disabled yes
set device-group LOCATIONS pre-rulebase security rules openphish-alert tag

 

This particular feed is using the openphish URL feed.

 

I can't attach a .txt file so that's why it was a pdf. Here is the dump of the edl WITHOUT the ?=panosurl flag.

 

BTW before I do this, do you really want me to paste >2000 lines into this thread? Here's a sneak preview:

 

vsys1/edl-phishing-sites:
Next update at : Fri Jan 4 09:00:36 2019
Source : https://10.X.X.X/feeds/phishing-url
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2108
Total invalid entries : 74
Valid urls:
http://venuesearch.in/include/santan/details.htm
http://venuesearch.in/include/santan/questions.htm
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/final.php
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/num.php
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://30dayaffiliatechallenge.com/wp-content/plugins/revslider/admin/views/system/.../china/?login=...
http://cesy.edu.mx/wp-admin/css/colors/chase/chase/home/auth/index.php
http://city-sm.ru/administrator/components/com_installer/helpers/mtch/match
http://d2gconsult.com.br/1c543ee7f3c60f6a7047d71b999da39dMzQzZjEzYjRmNmQ2MzNmYmJkZDMyNjc2NGNlMjRjMTM...
http://imroadrunner.com/dev/www.loginalibaba.com/alibaba/alibaba/login.alibaba.com.php?email=anko@an...
http://ledutech.org.br/logs/js/fix/f8870
http://msrebeco.cl/A1/dl/DHL.htm
http://nygift19.com/atb/index.html
http://nygift19.com/atb/questions.html
http://nygift19.com/bnc/National%20Bank%20Online.html
http://nygift19.com/cibc/index.php
http://nygift19.com/cibc/question.html
http://seediest-aids.000webhostapp.com/2/authws/1/login.php
http://venuesearch.in/include/santan/ssanta.htm

 

 

Hi @craigomatic,

would you mind sending me the .txt file over at lmori@paloaltonetworks.com?

I am trying to reproduce the issue, checking if it is an issue on MM or in PAN-OS.

 

Luigi

Sure it's on the way. Also included the MineMeld config.

Hi Luigi,

 

Did you get everything you need to reproduce the error? Let me know and I can provide you with my config. Thanks

Hi @craigomatic,

I haven't received the email with indicator list, could you please send it again please?

 

Luigi

OK I sent it ... please let me know if you do NOT receive. There are four attachments but they are small txt files.

Hi @craigomatic,

sorry, but I didn't receive it again. Are you on the community slack channel? Or could you send me an email without attachments first?

Hi @craigomatic,

just tested it and I can't replicate so far. Please could you tell me:

  • PAN-OS release
  • How the policy looks like (app/service/URL Category or URL Filtering profile?)

 

Thanks!

Luigi

  • 14684 Views
  • 21 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!