Blocking TLDs with a URL filter

Reply
L1 Bithead

Blocking TLDs with a URL filter

Hello all,

 

I'm attempting to block about 1340 TLDs with a URL filter.  However, I can't seem to get the URL filter to not block any URL where the TLD string is part.

 

For example:

If I want to block the .able TLD, I block "*.able" via a URL Category that's linked to a URL filter that's linked to a profile on a policy. 

 

I expect the following results:

 

What actually happens:

 

So, the wildcard "*.able" acts the same as a regex .*able.*

 

 

You might think that "*.able/" resolves this in this case:

I expect to happen:

 

 

What actually happens:

So, the wildcard "*.able/" acts the same as a regex .*[.]able$

 

The reason why I wish to block TLDs is simple:
I ran a regressive query against all URLs accessed by my company for three months (we are capturing traffic using Moloch "network VCR") and only about 50 TLDs are ever accessed. ICANNs total list of TLDs contains 1405 TLDs (https://www.icann.org/resources/pages/tlds-2012-02-25-en).

 

The only time I see hits against odd TLDs is in attack events. So, given that the cost-reward is so high, I wish to block TLDs.

 

 

This seems like an odd weakness to have in the URL filtering engine that could be resolved with a single line of code.  So I'll hope and assume that I'm doing something wrong.

 

 

Is it possible to block TLDs?

 

SEs, please feel free to access case: 00515922

 

 

Thanks,

 

Matt

Community Manager

Re: Blocking TLDs with a URL filter

Hi Matt

 

the URL filtering manual entries are actually sort of regex rather than simple wildcards, so using .able would do the job

 

alternatively, if you create a custom threat signature, you can use more complex regex to match the TLDs


Help the community: Like helpful comments and mark solutions
Reaper out
L1 Bithead

Re: Blocking TLDs with a URL filter

I have blocked ".able".


What I (and you (?)) expect to happen:

block: nic.able
not block: www.able.org/index.html or foo.docs.able.google.com or https://encrypted.google.com/search?hl=en_q=inurl:able

What actually happens:

block: nic.able
not block: www.able.org/index.html or foo.docs.able.google.com and https://encrypted.google.com/search?hl=en_q=inurl:able


However, if I also block ".google":
What I (and you (?)) expect to happen:

block: nic.google
not block: www.google.com/index.html or docs.google.com or https://encrypted.google.com/search?hl=en&q=inurl%3Agoogle

What actually happens:

block: nic.google and https://encrypted.google.com/search?hl=en&q=inurl%3Agoogle
not block: www.google.com/index.html or docs.google.com

 

So, although .able worked, the strategy in general doesn't work.

 

Any further assistance is appreciated.

 

 

Thanks,

 

Matt

L7 Applicator

Re: Blocking TLDs with a URL filter

I've been playing around with this internally for a while and found that while blocking .able or .google works better then doing a *.able it's far from a perfect solution. I've had more luck with creating a custom threat signature however, the time that is needed to create a custom threat signature and properly test it before actaully deploying it is a much longer process in general.  

L1 Bithead

Re: Blocking TLDs with a URL filter

Thanks for the reply!

 

Do you have an example you're willing to share?

 

 

Thanks,

 

Matt

L2 Linker

Re: Blocking TLDs with a URL filter

Recently encountered the exact same issue regarding TLD blocks. Can anyone from PAN comment on why this is happening? According to the docs, there is an implied /* at the end of an entry. If that's the case, the entry *.download should not block *.download.windowsupdate.com, but it does. Like @mbrownnyc mentioned, manually adding the /* doesn't resolve the issue.
Community Manager

Re: Blocking TLDs with a URL filter

hi @jhopple

 

I went looking for that article as it is incorrect, I've updated it to reflect that actual situation (this was an article from 2009, apologies for the confusion)

 

custom URL entries match tokens, devided by separators

 

this entry in the admin guide describes it best: https://www.paloaltonetworks.com/documentation/80/pan-os/web-interface-help/objects/objects-security...


Help the community: Like helpful comments and mark solutions
Reaper out
L2 Linker

Re: Blocking TLDs with a URL filter

@reaper

Something you mentioned in a similar thread seems applicable here. Below is a summary of the solution you provided.

By adding ".com/" to the block list you will:
Block all domains with a .com tld
Still be able to access www.com.uk

(Ref. https://live.paloaltonetworks.com/t5/Management-Articles/URL-filtering-with-domain-name-patterns/tac...)

The solution implies the following:

If you want to block the .com tld
Then you should add "*.com/" to the block list
If you add "*.com/" to the block list
Then www.com.uk will not be blocked

Following that logic:

If "*.com/" blocks the .com tld
Then "*.download/" blocks the .download tld
If "*.com/" does not block www.com.uk
Then "*.download/" does not block download.windowsupdate.com

That means your solution should solve my problem.

Decided to double check the admin guide which says,

"Further, to block access to a domain suffix such as paloaltonetworks.com.au, you must add an entry with a slash ( / ) at the end. In this example, you would add *.paloaltonetworks.com/ to the block list."

That implies:

If you want to block paloaltonetworks.com.au
Then you must add "*.paloaltonetworks.com/" to the block list
If "*.paloaltonetworks.com/" will block paloaltonetworks.com.au
Then "*.paloaltonetworks.com/" will block paloaltonetworks.com.uk
If "*.paloaltonetworks.com/" will block paloaltonetworks.com.uk
Then "*.www.com/" will block www.com.uk
If "*.www.com/" will block www.com.uk
Then "*.*.com/" will block www.com.uk

So now we're getting into nested wildcards so I refered to a Management Article that says,

"The asterisk (*) wildcard does not respect the period (.) as a delimiter and will continue as a wildcard until a subdomain, domain or top level domain is matched.
*.*.sub3.com will match sub1.sub2.sub3.com. However, this should be avoided as a best practice as nested asterisks can create a performance impact on the device.
Instead, as a best practice you can use: *.sub3.com. This will match sub1.sub2.sub3.com"

(Ref. https://live.paloaltonetworks.com/t5/Management-Articles/Nested-Wildcard-in-URLs-May-Severely-Affect...)

So, picking up where we left off and keeping that in mind:

If "*.www.com/" will block www.com.uk
Then "*.*.com/" will block www.com.uk
If "*.*.com/" will block www.com.uk
Then "*.com/" will block www.com.uk
If "*.com/" will block www.com.uk

I ran this thing to the ground on my scratch pad but I'll stop here. At this the assumed logic from the admin guide contradicts the assumed logic of your solution from the other thread.

I would very much like to know:
If your solution is actually valid
If there was a typo in the admin guide
Community Manager

Re: Blocking TLDs with a URL filter

 if not for the slash at the end of the string you would be right. the slash is a hard stop to the domain name and will only allow url path after that

 

*.com/

blocks www.mysite.com

allows www.mysite.com.uk

 

*.com 

blocks www.mysite.com

blocks www.mysite.com.uk

 

i reproduced it real quick:

slash.pngslashnoslash.pngno slash

 

always trust @reaper ;)

 

I've pinged the admin guide team to check the wording


Help the community: Like helpful comments and mark solutions
Reaper out
L2 Linker

Re: Blocking TLDs with a URL filter

@reaper

That's kind of the direction I was leaning (ref typo on guide) That's also how I thought the trailing / was interpreted (ref hard stop). I still felt inclined to write out the logic (based on the doc said) even though it seemed ridiculous. For a brief moment I thought maybe y'all had a magical parser that could distinguish between a domain/sub-domain and a tld. Lasted about as long as it took to write out the first reference to ".download".

So now the only question I have is this: If I request approval to have "*.domain/" added to block list, can I use you as an authoritative source and if download.windowsupdate.com gets blocked are you willing to take the heat?


(That last bit is joke)
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!