Adding filter "?v=panosurl" broken access to all websites

[dannadurai]

Custom URL category is configured to block phishing URLs collected from Linux MineMeld server through EDL. For some reason adding filter "?v=panosurl" (https://10.9.0.60/feeds/phishing-url?v=panosurl) to retrieve URLs in PAN-OS supported format (malware.com) is creating issue as all the websites are categorized as phishing and blocked. Using without filter ( https://10.9.0.60/feeds/phishing-url) don't work because URLs are retrieved in format (http://malware.com)

Found this live community post for similar issue https://live.paloaltonetworks.com/t5/General-Topics/Adding-v-panosurl-to-MineMeld-EDL-brought-down-o...

What is the solution for this ?

[lmori]

Hi @dannadurai,

could you share which Miners are you using?

Luigi

[dannadurai]

Hello,

Using openphish miner

https://openphish.com/feed.txt

[dannadurai]

Thanks for the post; although I found it after I had experienced the same thing; however, my list did not include a *.com or *.it.

name@fw(active)>  request system external-list show type ip name edl-phishing-sites 
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

craigpdce

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

[craigomatic]

I was able to reproduce this at will on several systems running PANOS 8.1.4.

name@fw(active)>  request system external-list show type ip name edl-phishing-sites 
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

Attached the output of the "request system external-list" command. Sorry came out really huge when I converted to pdf.

[lmori]

Hi @craigomatic,

thanks, that was my same result (i.e. no URLs like *.<tld>). Let me look into this.

[lmori]

Hi @craigomatic,

I need some help in reproducing this issue, could you:

1. tell me about the policy you are using to enforce the phishing list? Are you using the EDL in URL profile or directly in the security policy under URL Category? Which application and service?
2. Could you share the dump of the feed instead of the pdf? Basically on the browser click on the URL and copy & paste the content in a plain text file
3. Are you using openphish or other feeds?

[craigomatic]

@lmori, thanks for your response.

I'm using the EDL URL category in the "category" field as follows:

set device-group LOCATIONS pre-rulebase security rules openphish-alert target negate no
set device-group LOCATIONS pre-rulebase security rules openphish-alert to untrust
set device-group LOCATIONS pre-rulebase security rules openphish-alert from [ trust ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert source any
set device-group LOCATIONS pre-rulebase security rules openphish-alert destination any
set device-group LOCATIONS pre-rulebase security rules openphish-alert source-user any
set device-group LOCATIONS pre-rulebase security rules openphish-alert category edl-phishing-sites
set device-group LOCATIONS pre-rulebase security rules openphish-alert application [ ssl web-browsing ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert service application-default
set device-group LOCATIONS pre-rulebase security rules openphish-alert hip-profiles any
set device-group LOCATIONS pre-rulebase security rules openphish-alert action drop
set device-group LOCATIONS pre-rulebase security rules openphish-alert log-setting lf-pnrm-logr-alienv
set device-group LOCATIONS pre-rulebase security rules openphish-alert disabled yes
set device-group LOCATIONS pre-rulebase security rules openphish-alert tag

This particular feed is using the openphish URL feed.

I can't attach a .txt file so that's why it was a pdf. Here is the dump of the edl WITHOUT the ?=panosurl flag.

BTW before I do this, do you really want me to paste >2000 lines into this thread? Here's a sneak preview:

vsys1/edl-phishing-sites:
Next update at : Fri Jan 4 09:00:36 2019
Source : https://10.X.X.X/feeds/phishing-url
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2108
Total invalid entries : 74
Valid urls:
http://venuesearch.in/include/santan/details.htm
http://venuesearch.in/include/santan/questions.htm
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/final.php
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/num.php
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.c...
http://30dayaffiliatechallenge.com/wp-content/plugins/revslider/admin/views/system/.../china/?login=...
http://cesy.edu.mx/wp-admin/css/colors/chase/chase/home/auth/index.php
http://city-sm.ru/administrator/components/com_installer/helpers/mtch/match
http://d2gconsult.com.br/1c543ee7f3c60f6a7047d71b999da39dMzQzZjEzYjRmNmQ2MzNmYmJkZDMyNjc2NGNlMjRjMTM...
http://imroadrunner.com/dev/www.loginalibaba.com/alibaba/alibaba/login.alibaba.com.php?email=anko@an...
http://ledutech.org.br/logs/js/fix/f8870
http://msrebeco.cl/A1/dl/DHL.htm
http://nygift19.com/atb/index.html
http://nygift19.com/atb/questions.html
http://nygift19.com/bnc/National%20Bank%20Online.html
http://nygift19.com/cibc/index.php
http://nygift19.com/cibc/question.html
http://seediest-aids.000webhostapp.com/2/authws/1/login.php
http://venuesearch.in/include/santan/ssanta.htm

[lmori]

Hi @craigomatic,

would you mind sending me the .txt file over at lmori@paloaltonetworks.com?

I am trying to reproduce the issue, checking if it is an issue on MM or in PAN-OS.

Luigi

[craigomatic]

Sure it's on the way. Also included the MineMeld config.

[craigomatic]

Hi Luigi,

Did you get everything you need to reproduce the error? Let me know and I can provide you with my config. Thanks

[lmori]

Hi @craigomatic,

I haven't received the email with indicator list, could you please send it again please?

Luigi

[craigomatic]

OK I sent it ... please let me know if you do NOT receive. There are four attachments but they are small txt files.

[lmori]

Hi @craigomatic,

sorry, but I didn't receive it again. Are you on the community slack channel? Or could you send me an email without attachments first?

[lmori]

Hi @craigomatic,

just tested it and I can't replicate so far. Please could you tell me:

• PAN-OS release
• How the policy looks like (app/service/URL Category or URL Filtering profile?)

Thanks!

Luigi