Security policy matches traffic, but custom URL category doesn't work

[Flang3r]

Hello,

We're running VM appliance with PAN-OS 10.1.4

I've created custom URL category matching several wcard domain names, that we want to whitelist for only some LDAP users and added security policy placing it above internet access policy for all users (where URL filtering profile is applied to block URLs by predefined categories). We're decrypting this traffic as well. Policy looks like this:

Src zone: Inside facing user networks
Src User: user1/user2/user3 (IP addresses obtained via querying PA user agent running in separate VM)
Dst zone: Outside facing Internet
Service: Any
Application: Any
Profile: None
URL Category: Custom URL category I've created
Action: Allow

This policy matches traffic generated from user1/2/3, but URLs are still blocked and notification indicates them being blocked by default category. Is there anything I'm missing?

[Adrian_Jensen]

Are the URLs blocked in the default category, but allowed in the custom category? Something I ran into before is that a URL Filtering Block always takes precedence over Allow, even if the Allow URL is more specific. 

[Flang3r]

Thanks for the reply.

Yes these URLs matched by custom URL category are blocked in default category assigned to URL filtering profile. But security policy rule, that uses this default category filtering profile is below (after) this specific rule. I've even tried making another URL filtering profile where I allowed this custom category, but even then they were blocked. There is no overlap of regex matches in single URL profile.

If block always takes precedence regardless its security rule position, then I can't understand how to allow portion of webpages to specific users/IPs, while blocking them to others. It's either all allow or all block then...

[Flang3r]

There is an odd thing I've noticed in logs. If this security rule above doesn't have custom URL category attached to it, matched traffic (by source LDAP user) is correctly allowed and forwarded, but once I attach URL category although it matches the URL regex, firewall continues evaluating security rules and finally blocks URL because it's blocked in rule below for all users.

[Adrian_Jensen]

So when you define a URL Category it can be used in 2 places:

 - a Service/URL Category traffic identifier under a Security Policy

 - in a URL Filtering object as an allow/block category (always appears in all URL Filtering objects defaulted to "none")

So you are using your custom URL Category in your specific allow Security Policy, but under the Actions tab of the policy where you set Allow, do you have a Profile Setting with a URL Filtering profile that is actually blocking the connection? Or perhaps the site is also matches a built in category that is set to block in the URL Filtering.

I.e.:

URL Category -> "Example":

    example.com/

    *.example.com/

URL Filtering -> "filterInternet"

    Example: block/block

    adult: block/block

    business: allow/allow

    government: allow/allow

    social-media: block/block

Policy -> Security -> "Allow general internet access"

    SrcZone = Trust

    DstZone = Untrust

    Action = Allow

      Profile-URL-Filtering = filterInternet   <= general block of example.com by category

Policy -> Security -> "Allow specific users to example.com"

    SrcZone = Trust

    SrcUser = Alice, Bob, Eve

    DstZone = Untrust

    URL Category = Example

    Action = Allow

      Profile-URL-Filtering = filterInternet   <= still blocking of example.com by category

So, for example, in our environment we block Facebook/Twitter/etc. by default using the build in social-networking category. We then have several custom URL categories where we define some Facebook/etc. domains and a custom URL Filter with those categories set to allow. Then, for a specific set of users we filter traffic in an addition allow rule using the alternate URL Filter. We don't try to identify the target URLs in the Security Policy itself.

Policy -> Security -> "Allow general internet access"

    SrcZone = Trust

    DstZone = Untrust

    Action = Allow

      Profile-URL-Filtering = filterInternet   <= general block of social-networking category

Policy -> Security -> "Allow specific users to example.com"

    SrcZone = Trust

    SrcUser = Alice, Bob, Eve

    DstZone = Untrust

    Action = Allow

      Profile-URL-Filtering = filterInternet-allowFacebook   <= general block of social-networking category with additional custom URL Category "Facebook-URLs" set to allow/allow

[Flang3r]

Hello, thank you again for your time.

So you are using your custom URL Category in your specific allow Security Policy, but under the Actions tab of the policy where you set Allow, do you have a Profile Setting with a URL Filtering profile that is actually blocking the connection?

True, I use rule with URL category in specific allow, but no profile is applied in that rule. Let's assume my URL category object name is "CAT1" and has following regex configured:

*.example.com/*

*.example.com/

.example.com/*

.example.com/

And URL filtering profile "PROF1", that has default categories either allowed or blocked, where "CAT1" is set to none. "example.com " is in one of the blocked default categories here.

Then I have two security rules as follows from up->down

Rule1 that allows several LDAP users to any destinations using any apps and any services, "CAT1" applied to it and profiles set to none.

Rule2 for "General Internet Access", that has profiles enabled and "PROF1" applied to it.

What I think I see in logs is, that CAT1 is matched before PROF1, but instead of allowing this traffic, firewall continues evaluating security policies down the line and finally blocks URL via PROF1.

I've also tried the scenario described by you using two separate URL filtering profiles, where in PROF1 CAT1 is set to "none/none" and in PROF2 (clone of PROF1) CAT1 is set to "allow/allow". Even then, firewall still blocks access from Rule2 via PROF1.

[Adrian_Jensen]

---

---

What I think I see in logs is, that CAT1 is matched before PROF1, but instead of allowing this traffic, firewall continues evaluating security policies down the line and finally blocks URL via PROF1.

I've also tried the scenario described by you using two separate URL filtering profiles, where in PROF1 CAT1 is set to "none/none" and in PROF2 (clone of PROF1) CAT1 is set to "allow/allow". Even then, firewall still blocks access from Rule2 via PROF1.

---

---

 Hmmm, yeah it sounds like you are on the right track, but the PA is taking the General Internet Access rule as a more specific match. Other than that you do not need both *.example.com/ and *.example.com/* in the category (doesn't cause a problem, just a duplicate), I don't see a problem and am a bit stumped.

[Flang3r]

I found what the problem was. It has to do with application IDs.

If you leave application to "any" and apply URL filtering profile with URL category, it only matches "web-browsing" and "ssl" app-ids for URL list regex. Anything else associated with that URL will be governed by predefined category containing it and get blocked. If you add the rest of app-ids exclusively (necessary for a webpage like wetransfer for example), then it will correctly allow traffic on that rule.

That's why when I was matching with "any" in application field, I could see hits on specific rule but also see blocking from generic rule. web-browsing and ssl were matched in specific rule, wetransfer-base etc., matched further down in generic. Seems like "any" doesn't really mean any in the world of PA...

Actually if URL requires only web-browsing and ssl app-ids it works even with URL category only, but there is a caveat. Rules with URL category only will ignore any other app-ids (even added exclusively) except web-browsing and ssl. Pretty convoluted I should say.

Thanks for helping me 🙂

[Kobiher]

---

@Flang3r wrote:

I found what the problem was. It has to do with application IDs.

If you leave application to "any" and apply URL filtering profile with URL category... If you add the rest of app-ids exclusively (necessary for a webpage like wetransfer for example), then it will correctly allow traffic on that rule..

[...]

Actually if URL requires only web-browsing and ssl app-ids it works even with URL category only, but there is a caveat. Rules with URL category only will ignore any other app-ids (even added exclusively) except web-browsing and ssl. Pretty convoluted I should say.

I know that it's been a year and more since this was written, but I find those two statements to be a contradiction to each other - If I understand this correctly, in the first one, you state that if "add the rest of the app-ids exclusively, then it will correctly allow traffic on that rule", but then in the second paragraph you state that "rules with URL category only will ignore any other app-ids (even added exclusively) " - so which is it and how can those two both be true ? what am I missing here ?