topic Re: Adding filter "?v=panosurl" broken access to all websites in General Topics

Adding filter "?v=panosurl" broken access to all websites

dannadurai — Tue, 01 Jan 2019 22:30:44 GMT

Custom URL category is configured to block phishing URLs collected from Linux MineMeld server through EDL. For some reason adding filter "?v=panosurl" (https://10.9.0.60/feeds/phishing-url?v=panosurl) to retrieve URLs in PAN-OS supported format (malware.com) is creating issue as all the websites are categorized as phishing and blocked. Using without filter ( https://10.9.0.60/feeds/phishing-url) don't work because URLs are retrieved in format (http://malware.com)

Found this live community post for similar issue https://live.paloaltonetworks.com/t5/General-Topics/Adding-v-panosurl-to-MineMeld-EDL-brought-down-our-entire/m-p/220352#M63569

What is the solution for this ?

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Wed, 02 Jan 2019 07:42:31 GMT

Hi @dannadurai,

could you share which Miners are you using?

Luigi

Re: Adding filter "?v=panosurl" broken access to all websites

dannadurai — Wed, 02 Jan 2019 18:47:51 GMT

Hello,

Using openphish miner

https://openphish.com/feed.txt

Re: Adding filter "?v=panosurl" broken access to all websites

dannadurai — Thu, 03 Jan 2019 18:59:08 GMT

Thanks for the post; although I found it after I had experienced the same thing; however, my list did not include a *.com or *.it.

name@fw(active)> request system external-list show type ip name edl-phishing-sites
vsys1/edl-phishing-sites:
Next update at : Thu Dec 27 16:00:02 2018
Source : https://10.x.x.x/feeds/phishing-url?v=panosurl
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2013
Total invalid entries : 59

craigpdce

Went through the entire text and did not find a string or consecutive wildcards together. Can't figure out why this would have recategorized pretty much every common domain as edl-phishing-sites. Thankfully I deny all traffic to those sites with my policy. We had a connectivity issue for about 5 minutes until I could back everything out. What a pita.

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Thu, 03 Jan 2019 19:09:03 GMT

I was able to reproduce this at will on several systems running PANOS 8.1.4.

Total valid entries : 2013
Total invalid entries : 59

Attached the output of the "request system external-list" command. Sorry came out really huge when I converted to pdf.

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Fri, 04 Jan 2019 08:33:26 GMT

Hi @craigomatic,

thanks, that was my same result (i.e. no URLs like *.<tld>). Let me look into this.

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Fri, 04 Jan 2019 09:26:54 GMT

Hi @craigomatic,

I need some help in reproducing this issue, could you:

tell me about the policy you are using to enforce the phishing list? Are you using the EDL in URL profile or directly in the security policy under URL Category? Which application and service?
Could you share the dump of the feed instead of the pdf? Basically on the browser click on the URL and copy & paste the content in a plain text file
Are you using openphish or other feeds?

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Fri, 04 Jan 2019 16:18:22 GMT

@lmori, thanks for your response.

I'm using the EDL URL category in the "category" field as follows:

set device-group LOCATIONS pre-rulebase security rules openphish-alert target negate no
set device-group LOCATIONS pre-rulebase security rules openphish-alert to untrust
set device-group LOCATIONS pre-rulebase security rules openphish-alert from [ trust ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert source any
set device-group LOCATIONS pre-rulebase security rules openphish-alert destination any
set device-group LOCATIONS pre-rulebase security rules openphish-alert source-user any
set device-group LOCATIONS pre-rulebase security rules openphish-alert category edl-phishing-sites
set device-group LOCATIONS pre-rulebase security rules openphish-alert application [ ssl web-browsing ]
set device-group LOCATIONS pre-rulebase security rules openphish-alert service application-default
set device-group LOCATIONS pre-rulebase security rules openphish-alert hip-profiles any
set device-group LOCATIONS pre-rulebase security rules openphish-alert action drop
set device-group LOCATIONS pre-rulebase security rules openphish-alert log-setting lf-pnrm-logr-alienv
set device-group LOCATIONS pre-rulebase security rules openphish-alert disabled yes
set device-group LOCATIONS pre-rulebase security rules openphish-alert tag

This particular feed is using the openphish URL feed.

I can't attach a .txt file so that's why it was a pdf. Here is the dump of the edl WITHOUT the ?=panosurl flag.

BTW before I do this, do you really want me to paste >2000 lines into this thread? Here's a sneak preview:

vsys1/edl-phishing-sites:
Next update at : Fri Jan 4 09:00:36 2019
Source : https://10.X.X.X/feeds/phishing-url
Referenced : Yes
Valid : Yes
Auth-Valid : Yes

Total valid entries : 2108
Total invalid entries : 74
Valid urls:
http://venuesearch.in/include/santan/details.htm
http://venuesearch.in/include/santan/questions.htm
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/final.php
http://fgs.ge/wp-admin/css/css/bp/2f10f/cyberplusauthentification/num.php
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.com/Logon.aspx/confirmyouridentity.php
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.com/Logon.aspx/unabletologon.html
http://etajerki.ru/sites/all/modules/ckeditor/ChaseBank/Chase%20Bank/securechase/chaseonline.chase.com/Logon.aspx/verifyyouraccount.html
http://30dayaffiliatechallenge.com/wp-content/plugins/revslider/admin/views/system/.../china/?login=jim@thejimburkefamily.com
http://cesy.edu.mx/wp-admin/css/colors/chase/chase/home/auth/index.php
http://city-sm.ru/administrator/components/com_installer/helpers/mtch/match
http://d2gconsult.com.br/1c543ee7f3c60f6a7047d71b999da39dMzQzZjEzYjRmNmQ2MzNmYmJkZDMyNjc2NGNlMjRjMTM=/myaccount/websc_login/?country.x=&locale.x=en_
http://imroadrunner.com/dev/www.loginalibaba.com/alibaba/alibaba/login.alibaba.com.php?email=anko@anko.com.tw
http://ledutech.org.br/logs/js/fix/f8870
http://msrebeco.cl/A1/dl/DHL.htm
http://nygift19.com/atb/index.html
http://nygift19.com/atb/questions.html
http://nygift19.com/bnc/National%20Bank%20Online.html
http://nygift19.com/cibc/index.php
http://nygift19.com/cibc/question.html
http://seediest-aids.000webhostapp.com/2/authws/1/login.php
http://venuesearch.in/include/santan/ssanta.htm

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Fri, 04 Jan 2019 16:05:02 GMT

Hi @craigomatic,

would you mind sending me the .txt file over at lmori@paloaltonetworks.com?

I am trying to reproduce the issue, checking if it is an issue on MM or in PAN-OS.

Luigi

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Fri, 04 Jan 2019 16:28:22 GMT

Sure it's on the way. Also included the MineMeld config.

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Thu, 10 Jan 2019 21:40:22 GMT

Hi Luigi,

Did you get everything you need to reproduce the error? Let me know and I can provide you with my config. Thanks

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Fri, 11 Jan 2019 08:13:31 GMT

Hi @craigomatic,

I haven't received the email with indicator list, could you please send it again please?

Luigi

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Fri, 11 Jan 2019 20:56:30 GMT

OK I sent it ... please let me know if you do NOT receive. There are four attachments but they are small txt files.

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Sun, 13 Jan 2019 08:49:16 GMT

Hi @craigomatic,

sorry, but I didn't receive it again. Are you on the community slack channel? Or could you send me an email without attachments first?

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Mon, 14 Jan 2019 17:28:17 GMT

Hi @craigomatic,

just tested it and I can't replicate so far. Please could you tell me:

PAN-OS release
How the policy looks like (app/service/URL Category or URL Filtering profile?)

Thanks!

Luigi

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Mon, 14 Jan 2019 17:52:41 GMT

Luigi,

PANOS release 8.1.4.

set device-group ACME external-list edl-phishing-sites type url recurring hourly
set device-group ACME external-list edl-phishing-sites type url certificate-profile minemeld_cert_profile
set device-group ACME external-list edl-phishing-sites type url url https://10.X.X.X/feeds/phishing-url

Issue can be replicated by changing the last line:

Then most/any url categories will be defined as edl-phishing-sites. I saw google.com, bing.com, yahoo.com being classified as such.

Here's the URL filtering profile, which includes the edl:

set device-group ACME profiles url-filtering ACME-PAN-URL-Policy credential-enforcement block [ abortion abused-drugs adult alcohol-and-tobacco auctions command-and-control copyright-infringement dynamic-dns extremism gambling games hacking home-and-garden hunting-and-fishing internet-communications-and-telephony malware nudity parked peer-to-peer phishing proxy-avoidance-and-anonymizers questionable weapons web-advertisements edl-phishing-sites shortened-urls ]

Here's a policy we were using specifically to block matching category:

craigomatic@PNRM01# show | match openphish-alert
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert target negate no
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert to untrust
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert from [ trust ]
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert source any
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert destination any
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert source-user any
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert category edl-phishing-sites
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert application [ ssl web-browsing ]
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert service application-default
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert hip-profiles any
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert action drop
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert log-setting lf-pnrm-siem
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert disabled no
set device-group ACME-LOCATIONS pre-rulebase security rules openphish-alert tag

Let me know if you need more info. I was able to reproduce this on my lab PA-220. I can ship you that config if that would be helpful.

Re: Adding filter "?v=panosurl" broken access to all websites

craigomatic — Mon, 28 Jan 2019 16:42:55 GMT

Luigi,

Just tried in 8.1.5 and was unable to reproduce error in my testing environment. I will roll out to production cautiously and will update ...

Re: Adding filter "?v=panosurl" broken access to all websites

lmori — Thu, 07 Feb 2019 08:11:52 GMT

Hi @craigomatic,

thanks, please let me know the outcome of your tests.

Re: Adding filter "?v=panosurl" broken access to all websites

Remo — Sun, 12 May 2019 16:29:48 GMT

Hi @lmori

Were you able to solve this issue? It is definately an issue in minemeld. After adding this parameter, for some reason "*.com" was on the output ...

Re: Adding filter "?v=panosurl" broken access to all websites

Remo — Sun, 12 May 2019 23:19:38 GMT

Currently the reason for this issue is a bug of minemeld that handles special entries the wrong way. I wrote the more detailled description here: https://live.paloaltonetworks.com/t5/General-Topics/Adding-v-panosurl-to-MineMeld-EDL-brought-down-our-entire/m-p/260646/highlight/true#M73886