Episode Transcript:
Hello PANCasters and welcome back! In today's episode, we are going to have a look at URL Filtering. This is one of the main security and policy features of our next-gen firewalls along with Prisma Access. It is also one that you should spend a little bit of time understanding, so that when you are configuring your policies, what you expect to allow or block is actually what happens.
Let’s start with a quick recap of URL Filtering. URL Filtering lets you control and monitor websites that users can access, based on either predefined URL categories or your own custom categories. URL Filtering has a lot of uses and features — like preventing credential phishing, multi-category filtering and safe search enforcement. Aside from controlling access, it can also be used for things like deciding which URLs or categories will be included or excluded from SSL decryption. We won’t go into the details of all the features, as these can be found on our admin guides. What we will discuss today is some of the things to be aware of in your configuration so your policy is working as you want it to.
John Arena is a Professional Services Consultant with a background in Technical Support for Palo Alto Networks and a passion for educating and sharing knowledge with customers.
First, let’s discuss pre-defined versus custom categories. Pre-defined is what we at Palo Alto Networks classify a URL to be related to. This is also known as PAN-DB. As an example, www.google.com is classified as a search engine. We also have a number of pre-defined categories you want to block, for example malware. As you can appreciate, the number of URLs worldwide is now incredibly huge, so we do use automation to help. If you do come across a URL that you think is misclassified, you can request it to be changed. You can use the Palo Alto Networks’ Test A Site to confirm the category and then request a change. If you notice this while checking your firewall logs, you can actually submit the request from the firewall web UI. Using these methods the relevant Palo Alto team that handles URL classification will review your request and then get back to you.
While this covers most of the requirements, there are always going to be exceptions, which is where custom categories come in. Let’s say you want to block all shareware and freeware sites so that users can’t download software from them. You do, however, need to allow a couple of sites that are under this classification. With a custom category you can allow only specific URLs and then block all other shareware and freeware sites using the pre-defined category. For this to work properly, the ordering and the configuration of your security policy is obviously important so, on that note, I want to discuss a key point about URL Filtering.
This is definitely easier to understand if you are looking at a firewall web UI, but you can get the overall idea now and always look at the UI later. URL filtering can be applied in two sections of your security policy: in the Service/URL policy match or as a security profile in the actions section. Let’s just go through these:
When you add either a pre-defined or custom URL category to the Service/URL category in the security policy, this is the same as adding a source IP or a service port. This is used for traffic to match that security policy. So, the same as a source IP, if the traffic does not match the URL you have specified it just continues down the security policy to find a match. The key here is the URL the client is requesting is not logged if the traffic does not even match that policy.
I have had quite a few instances where customers say URL Filtering is not working, and they can’t see why because they don’t see any sites being blocked. These were because a custom category was used in the policy match and in most cases, the website was calling a number of different URLs which were not all added to the custom category. This meant the site was not displaying properly. If this is the case, you will need to do some additional troubleshooting. The easiest way would be to use developer tools in your browser. Most browsers have this and it can be used to get the details on what the browser is actually requesting when you go to a site. Using developer tools, you should be able to see any additional URLs that are being requested and perhaps being blocked.
The second way of configuring URL filtering is on the security profile. When this is configured in the security profile then unlike a policy match, this is applying security processing on that traffic. This is when things like logging, safe search and credential phishing can be applied. It is good to understand the difference between where you configure URL filtering, as a policy match or as a security profile. While there are valid reasons for both, the security profile is more common as this relates to the actual security processing for URL filtering.
There’s another key thing to understand about URL filtering. By default, the action on allowed URL categories is an action named “allow”. This allows the traffic, but does not log the URL. This is commonly known now but I have still seen a few instances where default profiles are used, so there is no URL logging for allowed traffic. The action named “alert” is the one used to log the allowed traffic. Look, there may be some scenarios where you don’t want to log allowed URLs or maybe you don’t want to log some specific categories, but generally these days visibility is key. You probably want to make sure you have set these to “alert” to get the logs. It is also pretty easy to change all of the “allow” to “alert” in the profile by searching for allow and then doing a bulk change.
OK. I have left perhaps one of the most important things until last. It shouldn’t take too long, so please stay with me for this bit: The syntax you use with custom URL categories is extremely important. Why? Because small details could be the difference between allowing or blocking a site that shouldn’t be.
I have seen this so many times. Fortunately, never where it has been a major issue, but it’s important to understand how the matching works. Before I get into this, don’t worry about remembering all the details as you can find this in our KBs (listed below), but the key is you remember that there are specifics on how matching works.
Let’s start with going through what options you have for matching custom URL categories. The obvious one is a standard URL like www.google.com. Pretty easy, you just add it as is. But what is more common is you want to allow all google subdomains. So, we can include things like maps.google.com and mail.google.com. That’s where wildcards come in and the two common ones we use are star and caret.
The star and the caret work differently and, depending what you want to achieve, one may be better suited. Remember the wildcard can also be added at the start in the middle or at the end of the URL; this also has a significant bearing on what will match. The thing here is to understand that a star is a wildcard that means any number of additional domains, but a caret means only one additional domain. When I talk about domains, I mean any word in front of or in between a full stop. Let’s go through an example. So *.google.com allows www.google.com, mail.google.com but also video-stats.video.google.com. Also because it has a star wildcard, this will not match just google.com. If your intention is just to allow domain x .google.com, then you would use a caret. The caret will only match one additional domain so, in this case, video-stats.video.google.com would not match with a caret.
Again, don’t worry about memorizing this, just remember there are differences and you can have a look in our admin guides for the details and examples.
So the key takeaways for today are:
There you have it. Another PANCast, and I hope this has been helpful with some specific details about URL Filtering from the field. The actual uses and configuration are pretty varied and we now have a lot of additional features tied in with URL Filtering, so I’d also suggest you have a look at the URL Filtering section on our admin guides (find resources below). There is some great information there on advanced URL Filtering, use cases, best practices and troubleshooting.
In the next episode we’ll be talking about dataplane CPU and reasons it can go high so make sure you subscribe or just keep an eye out for the new episode.
Check out the full YouTube playlist: PANCast: Insights for Your Cybersecurity Journey.
Related Content:
