Blocking Web Advertisements with an External Dynamic List

[KevinTucker]

Hello everyone,

I am attempting to block web advertisements on our PA-3020. We have two of these devices which utilize Panorama. We have blocked anything categorized as "web-advertisement" on the firewall, which is great, but a ton of ads are still getting through. What we would like to do is as follows:

• Utilize an external dynamic list (a text file) to block domains
• When a blocked site is visited, the page will resolve to an "ad blocked" page rather than just not resolving the page at all.

Does anyone have any suggestions we can use? For our typical malware blacklisting, the sites just do not resolve and show a "page cannot be displayed" message. This is logged, which is great, but for the web advertisement blocking, our CIO wants it to show an "Ad Blocked" message. This message works for anything classified as "web-advertisement."

Any suggestions would be greatly appreciated.

Thanks,

Kevin

[BPry]

To the best of my knowledge, if you are using an EBL then you will simply 'deny' the connections from being made. I don't believe that PA has a 'denied' responce page in the context of denying hosts from the network, you can only get that with application response pages.

It might be a better option to look into rolling out ad-blocking via group policy. There are plenty of guides that you can follow and it doesn't take long at all, you also get the added benefit of disabling it when the user wants to access a site that would have otherwise denied access if it couldn't reach it's ad servers. 

[KevinTucker]

Thanks for the reply, BPry. I had it configured previously to just block the sites, however, without showing the "ad blocked" message, our CIO wasn't pleased with that. The GPO is managed by our large company, so it would be worth reaching out to them I believe. I appreciate your input very much.

[Brandon_Wertz]

It's a rather blunt method, but depending on how many domains you're populating in the EBL can you just not create your own local DNS poisoning?

Stand up a local internal page with the required block page and for ever domain just put a DNS entry on your network to point hosts to that internal page?

[birkhojk · Accepted solution]

One great way I could think of doing this is to go under Objects, External Dynamic Lists... and create a new Dynamic Domain Lists via Add and selecting Type = Domain List.  No doubt, you put the URL and the frequency it updates.

Next to enable it...

Security Profiles, Anti-Spyware, and open the DNS Signatures tab.

In there, you can apply an External Dynamic LIst Domains... I personally recommend "singhole"

**********

Now it is just a matter of going to your policy that lets your Internet traffic go out and changing the profile to that Anti Spyware profile.  Personally, I would recommend doing a Security Profile Group, so you can have consistency.throughout all of your Policies that do filtering via some preconfigured templates that you make.

+++++++++++++++++

Another option is to create a URL Block List... Same as aboe only you apply it to the URL Filtering under Security Profile.  It basically shows up with a little "+" next to it.  Naturally, you would need to change it in your active URL policy to an action of "Block"

**************************

Another way is to create an IP block list... Again it is in the External Dynamic Lists.

You would generally apply these via a security policy before your Internet policy.  Might make a policy that says something like your normal Inside TO Outside (Destination Address Dynamic IP List YOURBLOCKLIST) ... DENY

Then it ends up in the firewall logs with that rule showing it dropped if the IP address is in the list.

No doubt, you could also use hte "Destination Negate" option in your Internet Out rule and simply only ALLOW Internet traffic that doesn't match an IP on a Dynamic IP List.

Hope that helps... there is a TON of flexibility with the Palo Alto to block ads.

______________________________________________

It depends on which list-type you use what your block page will look like.

For example, blocking IPs more or less simply show up in the log.  If you do this, you want to reset-client or reset-both... otherwise the browser will just hang a while before timing out its TCP session, but you don't really get a block message.

If you are doing a Domain Block, that will give you your Antivirus / Anti-Spyware Block Page...

If you are doing an Extra Dynamic List on the URL Filtering, which is what you most likely want to do then it would use your normal URL Filtering block page.

[mst]

BTW, Kevin could You share what source of ad domains do You use?

[mivaldi]

Check this one out: https://pgl.yoyo.org/as

EDL as a Domain list:

https://pgl.yoyo.org/as/serverlist.php?hostformat=plain&showintro=0&startdate%5Bday%5D=&startdate%5B...

EDL as an URL list:

https://pgl.yoyo.org/as/serverlist.php?hostformat=adblock;showintro=0&startdate%5Bday%5D=&startdate%...

[Mohamed_Mabrouk]

Hi,

I tried the EDL with the follwoing link :

https://pgl.yoyo.org/as/serverlist.php?hostformat=plain&showintro=0&startdate%5Bday%5D=&startdate%5B...

but this list dosen`t have starts for domains like " *.hostname.com" , I tried another list that have *.xyz.com but it didn`t work

it give me that the EDL is not vaild.

Is the EDL supports Starts ( ex:*.xyz.com) ??

any suggestions ?

[mst]

@Mohamed_Mabrouk: which version of PAN-OS You have installed? Before 7.1 You can use only IP addresses and ranges in EBL, IMHO (look at this https://live.paloaltonetworks.com/t5/PAN-OS-7-1-Videos/PAN-OS-7-1-URL-Filtering-Dynamic-Block-List-E...).

[Mohamed_Mabrouk]

Hello,

I am using Paloalto 7.1, I added the same EDL but with Plaintext and it works well

This the list that I used , it contains *.xyz.com

https://pgl.yoyo.org/as/serverlist.php?hostformat=adblock;showintro=0&startdate%5Bday%5D=&startdate%...

List format requirements

• List must be a plain text document (no HTML, no PDF, etc.).
• Scheme is optional, and will be truncated if found – even if it is incomplete.
• http:// is not needed.
• Wildcards (*) are supported.
  • *.example.com/hacked/*
  • www.example.net/wp-*/debug/
• Maximum length per line is 1024 characters.
• Double-byte characters not supported.
• If specifying a domain, use both formats (as with custom URL categories):
  • example.com
  • *.example.com

 https://live.paloaltonetworks.com/t5/PAN-OS-7-1-Videos/PAN-OS-7-1-URL-Filtering-Dynamic-Block-List-E...

Thanks 

[mst]

nice 🙂
so i'll try it on 8.0beta .

[mivaldi]

You can use the Yoyo lists in two EDL types, Domain and URL.

With the Domain type, you can block access to the Ads when there is a DNS query to resolve the IP of the server hosting the Ads.

With the URL type, you can block access to the Ads when the host already has a cached IP for the domain, and submits a Client Hello with a matching SNI, or matching the HTTP GET. For HTTPS sessions you need to combine this solution with SSL Decryption to be able to pick up on the HTTP GET message inside the encrypted session.

The way that I implemented this is:

EDL as a Domain list 'Yoyo Ads - Domains':

https://pgl.yoyo.org/as/serverlist.php?hostformat=plain&showintro=0&startdate%5Bday%5D=&startdate%5B...

EDL as an URL list 'Yoyo Ads - URL':

https://pgl.yoyo.org/as/serverlist.php?hostformat=adblock;showintro=0&startdate%5Bday%5D=&startdate%...

I configured 'Yoyo Ads - Domains' under my Anti-Spyware profile to block:

Note that the drawback of this solution is that the Threat logs will still get filled up with 'Suspicious DNS query' drop alerts for TID 12000000 - I couldn't find a way to create a logging exception.

The next step was to configure a Security Policy that precedes my 'Internet access' rule to block any Client Hello messages containing a matching SNI. This will take care of HTTPS sessions that got as far as resolving to an IP and attempting to initiate an SSL session to the Ad servers in the list.

You could remove the checkbox for 'Log at Session End' to reduce logging in the Traffic logs if you're not interested in logging for this.

Finally the last step is, if any clear text or decrypted HTTP makes it through, we block the Ad with URL filtering, which profile is tied to our Internet access security rule.

Note that this should be combined with SSL decryption to extend coverage for encrypted HTTP(S) traffic. To make SSL Decryption effective for Chrome browsers, configure a security policy that precedes these rules to Deny 'quic' traffic.

Finally my Security policy set looks like this:

(The Sinkhole rule ties to the sinkhole action for Palo Alto Networks DNS Signatures in the Anti-Spyware profile, you can alternatively choose to sinkhole your 'Yoyo Ads - Domains', but as a result, that will mix your Traffic log entries for compromised host detection, as both, an infected host, or a host browsing to an advertisement will result in a 'subsequent' connection to the Sinkhole IP - so instead - using block for Ads will help you prevent the unintended Traffic log pollution).

If you have better ways to implement this please feel free to pitch in.

[tsheldon]

Mivaldi,

I set this up as you describe and it appears to be working great. I see the denies in the various logs. 

Your post is over a year old. Do you have any words of wisdom to add to it since then?

My commit alert is telling me I need to enable certificates. Will look into that next.

To

[mivaldi]

With the release of PAN-OS 8.0, there are a couple things to add. Even though you won't be able to except TID 12000000 from writing to the Threat Logs, you can actually define a log forwarding filter to prevent these from propagating to Panorama or Splunk (Syslog).

For a tutorial see:

https://live.paloaltonetworks.com/t5/Tutorials/Tutorial-Filtered-Log-Forwarding/ta-p/145950

Also, if there happens to be any domain that needs to be excepted, you can now search and Except domains directly from the EDL's.

[tsheldon]

Very good. Thank you for posting this information!