topic Re: Blocking Web Advertisements with an External Dynamic List in General Topics

Blocking Web Advertisements with an External Dynamic List

KevinTucker — Mon, 26 Sep 2016 14:56:54 GMT

Hello everyone,

I am attempting to block web advertisements on our PA-3020. We have two of these devices which utilize Panorama. We have blocked anything categorized as "web-advertisement" on the firewall, which is great, but a ton of ads are still getting through. What we would like to do is as follows:

Utilize an external dynamic list (a text file) to block domains
When a blocked site is visited, the page will resolve to an "ad blocked" page rather than just not resolving the page at all.

Does anyone have any suggestions we can use? For our typical malware blacklisting, the sites just do not resolve and show a "page cannot be displayed" message. This is logged, which is great, but for the web advertisement blocking, our CIO wants it to show an "Ad Blocked" message. This message works for anything classified as "web-advertisement."

Any suggestions would be greatly appreciated.

Thanks,

Kevin

Re: Blocking Web Advertisements with an External Dynamic List

BPry — Mon, 26 Sep 2016 15:05:00 GMT

To the best of my knowledge, if you are using an EBL then you will simply 'deny' the connections from being made. I don't believe that PA has a 'denied' responce page in the context of denying hosts from the network, you can only get that with application response pages.

It might be a better option to look into rolling out ad-blocking via group policy. There are plenty of guides that you can follow and it doesn't take long at all, you also get the added benefit of disabling it when the user wants to access a site that would have otherwise denied access if it couldn't reach it's ad servers.

Re: Blocking Web Advertisements with an External Dynamic List

KevinTucker — Mon, 26 Sep 2016 17:55:25 GMT

Thanks for the reply, BPry. I had it configured previously to just block the sites, however, without showing the "ad blocked" message, our CIO wasn't pleased with that. The GPO is managed by our large company, so it would be worth reaching out to them I believe. I appreciate your input very much.

Re: Blocking Web Advertisements with an External Dynamic List

Brandon_Wertz — Mon, 26 Sep 2016 18:20:16 GMT

It's a rather blunt method, but depending on how many domains you're populating in the EBL can you just not create your own local DNS poisoning?

Stand up a local internal page with the required block page and for ever domain just put a DNS entry on your network to point hosts to that internal page?

Re: Blocking Web Advertisements with an External Dynamic List

birkhojk — Fri, 30 Sep 2016 19:53:04 GMT

One great way I could think of doing this is to go under Objects, External Dynamic Lists... and create a new Dynamic Domain Lists via Add and selecting Type = Domain List. No doubt, you put the URL and the frequency it updates.

Next to enable it...

Security Profiles, Anti-Spyware, and open the DNS Signatures tab.

In there, you can apply an External Dynamic LIst Domains... I personally recommend "singhole"

**********

Now it is just a matter of going to your policy that lets your Internet traffic go out and changing the profile to that Anti Spyware profile. Personally, I would recommend doing a Security Profile Group, so you can have consistency.throughout all of your Policies that do filtering via some preconfigured templates that you make.

+++++++++++++++++

Another option is to create a URL Block List... Same as aboe only you apply it to the URL Filtering under Security Profile. It basically shows up with a little "+" next to it. Naturally, you would need to change it in your active URL policy to an action of "Block"

**************************

Another way is to create an IP block list... Again it is in the External Dynamic Lists.

You would generally apply these via a security policy before your Internet policy. Might make a policy that says something like your normal Inside TO Outside (Destination Address Dynamic IP List YOURBLOCKLIST) ... DENY

Then it ends up in the firewall logs with that rule showing it dropped if the IP address is in the list.

No doubt, you could also use hte "Destination Negate" option in your Internet Out rule and simply only ALLOW Internet traffic that doesn't match an IP on a Dynamic IP List.

Hope that helps... there is a TON of flexibility with the Palo Alto to block ads.

______________________________________________

It depends on which list-type you use what your block page will look like.

For example, blocking IPs more or less simply show up in the log. If you do this, you want to reset-client or reset-both... otherwise the browser will just hang a while before timing out its TCP session, but you don't really get a block message.

If you are doing a Domain Block, that will give you your Antivirus / Anti-Spyware Block Page...

If you are doing an Extra Dynamic List on the URL Filtering, which is what you most likely want to do then it would use your normal URL Filtering block page.

Re: Blocking Web Advertisements with an External Dynamic List

mst — Sun, 09 Oct 2016 11:12:24 GMT

BTW, Kevin could You share what source of ad domains do You use?

Re: Blocking Web Advertisements with an External Dynamic List

mivaldi — Tue, 31 Jan 2017 18:08:46 GMT

Check this one out: https://pgl.yoyo.org/as

EDL as a Domain list:

https://pgl.yoyo.org/as/serverlist.php?hostformat=plain&showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext

EDL as an URL list:

https://pgl.yoyo.org/as/serverlist.php?hostformat=adblock;showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext

Re: Blocking Web Advertisements with an External Dynamic List

Mohamed_Mabrouk — Tue, 31 Jan 2017 08:02:07 GMT

Hi,

I tried the EDL with the follwoing link :

https://pgl.yoyo.org/as/serverlist.php?hostformat=plain&showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext

but this list dosen`t have starts for domains like " *.hostname.com" , I tried another list that have *.xyz.com but it didn`t work

it give me that the EDL is not vaild.

Is the EDL supports Starts ( ex:*.xyz.com) ??

any suggestions ?

Re: Blocking Web Advertisements with an External Dynamic List

mst — Tue, 31 Jan 2017 08:26:23 GMT

@Mohamed_Mabrouk: which version of PAN-OS You have installed? Before 7.1 You can use only IP addresses and ranges in EBL, IMHO (look at this https://live.paloaltonetworks.com/t5/PAN-OS-7-1-Videos/PAN-OS-7-1-URL-Filtering-Dynamic-Block-List-External-Block-List/ta-p/74098).

Re: Blocking Web Advertisements with an External Dynamic List

Mohamed_Mabrouk — Tue, 31 Jan 2017 09:33:52 GMT

Hello,

I am using Paloalto 7.1, I added the same EDL but with Plaintext and it works well

This the list that I used , it contains *.xyz.com

https://pgl.yoyo.org/as/serverlist.php?hostformat=adblock;showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext

List format requirements

List must be a plain text document (no HTML, no PDF, etc.).
Scheme is optional, and will be truncated if found – even if it is incomplete.
http:// is not needed.
Wildcards (*) are supported.
- *.example.com/hacked/*
- www.example.net/wp-*/debug/
Maximum length per line is 1024 characters.
Double-byte characters not supported.
If specifying a domain, use both formats (as with custom URL categories):
- example.com
- *.example.com

https://live.paloaltonetworks.com/t5/PAN-OS-7-1-Videos/PAN-OS-7-1-URL-Filtering-Dynamic-Block-List-External-Block-List/ta-p/74098

Thanks

Re: Blocking Web Advertisements with an External Dynamic List

mst — Tue, 31 Jan 2017 11:58:08 GMT

nice 🙂
so i'll try it on 8.0beta .

Re: Blocking Web Advertisements with an External Dynamic List

mivaldi — Fri, 20 Apr 2018 16:47:06 GMT

You can use the Yoyo lists in two EDL types, Domain and URL.

With the Domain type, you can block access to the Ads when there is a DNS query to resolve the IP of the server hosting the Ads.

With the URL type, you can block access to the Ads when the host already has a cached IP for the domain, and submits a Client Hello with a matching SNI, or matching the HTTP GET. For HTTPS sessions you need to combine this solution with SSL Decryption to be able to pick up on the HTTP GET message inside the encrypted session.

The way that I implemented this is:

EDL as a Domain list 'Yoyo Ads - Domains':

https://pgl.yoyo.org/as/serverlist.php?hostformat=plain&showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext

EDL as an URL list 'Yoyo Ads - URL':

https://pgl.yoyo.org/as/serverlist.php?hostformat=adblock;showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext

I configured 'Yoyo Ads - Domains' under my Anti-Spyware profile to block:

Note that the drawback of this solution is that the Threat logs will still get filled up with 'Suspicious DNS query' drop alerts for TID 12000000 - I couldn't find a way to create a logging exception.

The next step was to configure a Security Policy that precedes my 'Internet access' rule to block any Client Hello messages containing a matching SNI. This will take care of HTTPS sessions that got as far as resolving to an IP and attempting to initiate an SSL session to the Ad servers in the list.

You could remove the checkbox for 'Log at Session End' to reduce logging in the Traffic logs if you're not interested in logging for this.

Finally the last step is, if any clear text or decrypted HTTP makes it through, we block the Ad with URL filtering, which profile is tied to our Internet access security rule.

Note that this should be combined with SSL decryption to extend coverage for encrypted HTTP(S) traffic. To make SSL Decryption effective for Chrome browsers, configure a security policy that precedes these rules to Deny 'quic' traffic.

Finally my Security policy set looks like this:

(The Sinkhole rule ties to the sinkhole action for Palo Alto Networks DNS Signatures in the Anti-Spyware profile, you can alternatively choose to sinkhole your 'Yoyo Ads - Domains', but as a result, that will mix your Traffic log entries for compromised host detection, as both, an infected host, or a host browsing to an advertisement will result in a 'subsequent' connection to the Sinkhole IP - so instead - using block for Ads will help you prevent the unintended Traffic log pollution).

If you have better ways to implement this please feel free to pitch in.

Re: Blocking Web Advertisements with an External Dynamic List

tsheldon — Fri, 20 Apr 2018 01:16:36 GMT

Mivaldi,

I set this up as you describe and it appears to be working great. I see the denies in the various logs.

Your post is over a year old. Do you have any words of wisdom to add to it since then?

My commit alert is telling me I need to enable certificates. Will look into that next.

Re: Blocking Web Advertisements with an External Dynamic List

mivaldi — Fri, 20 Apr 2018 16:37:27 GMT

With the release of PAN-OS 8.0, there are a couple things to add. Even though you won't be able to except TID 12000000 from writing to the Threat Logs, you can actually define a log forwarding filter to prevent these from propagating to Panorama or Splunk (Syslog).

For a tutorial see:

https://live.paloaltonetworks.com/t5/Tutorials/Tutorial-Filtered-Log-Forwarding/ta-p/145950

Also, if there happens to be any domain that needs to be excepted, you can now search and Except domains directly from the EDL's.

Re: Blocking Web Advertisements with an External Dynamic List

tsheldon — Mon, 23 Apr 2018 21:12:45 GMT

Very good. Thank you for posting this information!