Blocking TLDs with a URL filter

[mbrownnyc]

Hello all,

I'm attempting to block about 1340 TLDs with a URL filter.  However, I can't seem to get the URL filter to not block any URL where the TLD string is part.

For example:

If I want to block the .able TLD, I block "*.able" via a URL Category that's linked to a URL filter that's linked to a profile on a policy. 

I expect the following results:

• block: nic.able
• not block: www.able.org/index.html or foo.docs.able.google.com

What actually happens:

• block: nic.able, www.able.org/index.html and foo.docs.able.google.com

So, the wildcard "*.able" acts the same as a regex .*able.*

You might think that "*.able/" resolves this in this case:

I expect to happen:

• block: nic.able
• not block: www.able.org/index.html or foo.docs.able.google.com or https://encrypted.google.com/search?hl=en_q=inurl:able

What actually happens:

• block: nic.able and https://encrypted.google.com/search?hl=en_q=inurl:able
• not block: www.able.org/index.html or foo.docs.able.google.com

So, the wildcard "*.able/" acts the same as a regex .*[.]able$

The reason why I wish to block TLDs is simple:
I ran a regressive query against all URLs accessed by my company for three months (we are capturing traffic using Moloch "network VCR") and only about 50 TLDs are ever accessed. ICANNs total list of TLDs contains 1405 TLDs (https://www.icann.org/resources/pages/tlds-2012-02-25-en).

The only time I see hits against odd TLDs is in attack events. So, given that the cost-reward is so high, I wish to block TLDs.

This seems like an odd weakness to have in the URL filtering engine that could be resolved with a single line of code.  So I'll hope and assume that I'm doing something wrong.

Is it possible to block TLDs?

SEs, please feel free to access case: 00515922

Thanks,

Matt

[reaper]

Hi Matt

the URL filtering manual entries are actually sort of regex rather than simple wildcards, so using .able would do the job

alternatively, if you create a custom threat signature, you can use more complex regex to match the TLDs

[mbrownnyc]

I have blocked ".able".

What I (and you (?)) expect to happen:

block: nic.able
not block: www.able.org/index.html or foo.docs.able.google.com or https://encrypted.google.com/search?hl=en_q=inurl:able

What actually happens:

block: nic.able
not block: www.able.org/index.html or foo.docs.able.google.com and https://encrypted.google.com/search?hl=en_q=inurl:able

However, if I also block ".google":
What I (and you (?)) expect to happen:

block: nic.google
not block: www.google.com/index.html or docs.google.com or https://encrypted.google.com/search?hl=en&q=inurl%3Agoogle

What actually happens:

block: nic.google and https://encrypted.google.com/search?hl=en&q=inurl%3Agoogle
not block: www.google.com/index.html or docs.google.com

So, although .able worked, the strategy in general doesn't work.

Any further assistance is appreciated.

Thanks,

Matt

[BPry]

I've been playing around with this internally for a while and found that while blocking .able or .google works better then doing a *.able it's far from a perfect solution. I've had more luck with creating a custom threat signature however, the time that is needed to create a custom threat signature and properly test it before actaully deploying it is a much longer process in general.  

[mbrownnyc]

Thanks for the reply!

Do you have an example you're willing to share?

Thanks,

Matt

[jhopple]

Recently encountered the exact same issue regarding TLD blocks. Can anyone from PAN comment on why this is happening? According to the docs, there is an implied /* at the end of an entry. If that's the case, the entry *.download should not block *.download.windowsupdate.com, but it does. Like @mbrownnyc mentioned, manually adding the /* doesn't resolve the issue.

[reaper]

hi @jhopple

I went looking for that article as it is incorrect, I've updated it to reflect that actual situation (this was an article from 2009, apologies for the confusion)

custom URL entries match tokens, devided by separators

this entry in the admin guide describes it best: https://www.paloaltonetworks.com/documentation/80/pan-os/web-interface-help/objects/objects-security...

[jhopple]

@reaper

Something you mentioned in a similar thread seems applicable here. Below is a summary of the solution you provided.

By adding ".com/" to the block list you will:
Block all domains with a .com tld
Still be able to access www.com.uk

(Ref. https://live.paloaltonetworks.com/t5/Management-Articles/URL-filtering-with-domain-name-patterns/tac...)

The solution implies the following:

If you want to block the .com tld
Then you should add "*.com/" to the block list
If you add "*.com/" to the block list
Then www.com.uk will not be blocked

Following that logic:

If "*.com/" blocks the .com tld
Then "*.download/" blocks the .download tld
If "*.com/" does not block www.com.uk
Then "*.download/" does not block download.windowsupdate.com

That means your solution should solve my problem.

Decided to double check the admin guide which says,

"Further, to block access to a domain suffix such as paloaltonetworks.com.au, you must add an entry with a slash ( / ) at the end. In this example, you would add *.paloaltonetworks.com/ to the block list."

That implies:

If you want to block paloaltonetworks.com.au
Then you must add "*.paloaltonetworks.com/" to the block list
If "*.paloaltonetworks.com/" will block paloaltonetworks.com.au
Then "*.paloaltonetworks.com/" will block paloaltonetworks.com.uk
If "*.paloaltonetworks.com/" will block paloaltonetworks.com.uk
Then "*.www.com/" will block www.com.uk
If "*.www.com/" will block www.com.uk
Then "*.*.com/" will block www.com.uk

So now we're getting into nested wildcards so I refered to a Management Article that says,

"The asterisk (*) wildcard does not respect the period (.) as a delimiter and will continue as a wildcard until a subdomain, domain or top level domain is matched.
*.*.sub3.com will match sub1.sub2.sub3.com. However, this should be avoided as a best practice as nested asterisks can create a performance impact on the device.
Instead, as a best practice you can use: *.sub3.com. This will match sub1.sub2.sub3.com"

(Ref. https://live.paloaltonetworks.com/t5/Management-Articles/Nested-Wildcard-in-URLs-May-Severely-Affect...)

So, picking up where we left off and keeping that in mind:

If "*.www.com/" will block www.com.uk
Then "*.*.com/" will block www.com.uk
If "*.*.com/" will block www.com.uk
Then "*.com/" will block www.com.uk
If "*.com/" will block www.com.uk

I ran this thing to the ground on my scratch pad but I'll stop here. At this the assumed logic from the admin guide contradicts the assumed logic of your solution from the other thread.

I would very much like to know:
If your solution is actually valid
If there was a typo in the admin guide

[reaper]

 if not for the slash at the end of the string you would be right. the slash is a hard stop to the domain name and will only allow url path after that

*.com/

blocks www.mysite.com

allows www.mysite.com.uk

*.com 

blocks www.mysite.com

blocks www.mysite.com.uk

i reproduced it real quick:

slashno slash

always trust @reaper 😉

I've pinged the admin guide team to check the wording

[jhopple]

@reaper

That's kind of the direction I was leaning (ref typo on guide) That's also how I thought the trailing / was interpreted (ref hard stop). I still felt inclined to write out the logic (based on the doc said) even though it seemed ridiculous. For a brief moment I thought maybe y'all had a magical parser that could distinguish between a domain/sub-domain and a tld. Lasted about as long as it took to write out the first reference to ".download".

So now the only question I have is this: If I request approval to have "*.domain/" added to block list, can I use you as an authoritative source and if download.windowsupdate.com gets blocked are you willing to take the heat?


(That last bit is joke)

[reaper]

if you manage to get download.microsoft.com blocked by adding *.domain/ I will come over personally! 😉

but i reproduced it real quick for you: 

[jhopple]

@reaper

Thanks