topic Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat in General Topics

Custom URL Category in security rule - traffic log shows allowed with "any" in URL Category field

herrmoss — Wed, 16 Aug 2017 15:21:54 GMT

I've read the articles about the processes that take place when analyzing traffic and understand that sometimes there could be an allow status when it seems there shouldn't be. However it also seems that if the traffic truly shouldn't be allowed there would be an associated log entry with some kind of denial.
In my case there is no associated denial and I'm would still like to know why this traffic seems to be allowed when apparently not matching my Custom URL Category.
Forgive me if I'm still just misunderstanding something about this.

Thanks.

Here's what I'm seeing in my logs:

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

jvalentine — Wed, 16 Aug 2017 15:27:43 GMT

What if you also create a URL Filtering Profile (call it Alert All URL), configured with "alert" for all categories, and attach it to this rule?

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

Remo — Wed, 16 Aug 2017 16:11:43 GMT

Hi @herrmoss

Because of the any I think the reason is one of the following:

You did not attach the custom URL category to the policy
Traffic hits a rule before your rule with the attached custom URL category

If you share some more screenshots (logs with column 'rule', security policy in question, custom URL category), we should be able to help 😉

Regards,

Remo

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

herrmoss — Wed, 16 Aug 2017 18:12:19 GMT

Thanks for the reply.

Unfortunately we don't have a URL Filtering license as we have another solution for that function.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

Remo — Wed, 16 Aug 2017 18:33:30 GMT

For that what you intended to do (at least what I understood) - blocking or allowing specific websites with a custom URL category directly referenced in the security policy, you don't need an URL filtering license. Even for the logging part, mentionned by @jvalentine, there are workarounds that you will see the URL logs, without the license (only a workaround, not a recommended way)

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

herrmoss — Wed, 16 Aug 2017 20:31:39 GMT

You are correct @Remo I only mentioned the licensing issue becuase @jvalentine mentioned creating a URL Filtering Profile and I believe I can't do that without a license.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

herrmoss — Wed, 16 Aug 2017 20:35:06 GMT

Thanks @Remo attaching more screenshots.

And to be clear, the Custom URL Category is showing up for most of the log entries, I'm just confused about why it's not showing up for some.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

jvalentine — Wed, 16 Aug 2017 20:45:19 GMT

I just tried with an unlicensed lab firewall. I was able to create a URL filtering profile, configure it to alert on my custom categories, add the URL filtering profile to a security policy, and commit the changes. There is a commit warning complaining about no URL license, but it's a successful commit.

The colum that says "any" is only populated by attaching a URL filtering profile. I'm guessing this is why you're not seeing anything in that column.

Go ahead and create a URL filtering profile w/ "alert" as the action for all categories (or alert on your custom categories, and allow on all the rest), and attach it to your security policy. Commit and test.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

herrmoss — Wed, 16 Aug 2017 21:00:46 GMT

I didn't realize that you could create a URL Filtering profile without a license. Not sure I would like getting that warning (I assume I would continue to get this on subsequent commits).

Again, just to be clear, I do get my URL Category showing up in the logs (I just filtered those out in my previous images). Here's another look at the logs without the filtering and an example of "any" circled.

Also, my main goal here is to be able to confidentally tell upper management that no traffic is being allowed to a destination that is not in my URL Category.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

jvalentine — Wed, 16 Aug 2017 22:03:19 GMT

Understanding your main goal is helpful. Looking at your security policy, I don't see all of the applications you're allowing. Since SSL is in the list, I'm assuming web-browsing is in the list too. If it is, then this may be happening:

Client PC sends SYN on TCP/80, firewall doesn't know what the application is yet, but since you have a rule that allows web-browsing on TCP/80, that packet is permitted to pass. At this point the firewall doesn't know what the application is, let alone the URL.

Server responds with SYN/ACK. At this point, firewall still doesn't know what the application is yet, but since there are rules that allow web-browsing on TCP/80, firewall will continue to permit the handshake and subsequent data packets until it can definitively match your policies.

Client responds with ACK. Firewall still doesn't know application or URL.

Client sends some request on TCP/80. This is the firewall's first opportunity to look at layer-7 data and start determining things like application and URL. Some applications are immediately recognizeable and the firewall can take action beginning with this packet. Other applications require a couple of packets in order to definitively identify the application.

What is likely happening is that the firewall allows the TCP/80 traffic, even identifies it as web-browsing, and then it attempts to match that traffic with your permit rules. If it matches, great. If not, it stops. The trick question is, how should the firewall log the traffic? Should it log the traffic as being denied (when some portion went through?)

Can you add the packets sent / packets received columns to your log view? That would be an interesting datapoint for the traffic that you've circled. Also add the "Session ID" column. From the CLI run a "show session id xxxxxxxx" for the traffic log and see what additional detail is available.

It would also be helpful to know the exact URL that generated this traffic log. I seem to remember a way to do this. It involves creating a URL Filtering Profile...

EDIT: I'd recommend reading up on application dependencies, as this may be part of the issue. Also, this (https://live.paloaltonetworks.com/t5/Featured-Articles/DotW-Blocked-traffic-has-an-allow-log/ta-p/72357) is another possibility.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

Remo — Thu, 17 Aug 2017 06:25:56 GMT

@jvalentine

It is possible to configure the url profile without a license and apply it to a policy ... but did yoz really get url logs?

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

herrmoss — Fri, 18 Aug 2017 17:54:30 GMT

Sorry for the delay in responding but I've been tied up with other things.
Also I will not be able to work on this for the next week.
When I can get back to this I will look at some of the things you've mentioned.

However in regard to your comment:
"What is likely happening is that the firewall allows the TCP/80 traffic, even identifies it as web-browsing, and then it attempts to match that traffic with your permit rules. If it matches, great. If not, it stops. The trick question is, how should the firewall log the traffic? Should it log the traffic as being denied (when some portion went through?)"

I would think that if the firewall is going to log this traffic as allowed because it got part way through the process, it should also have a denied entry when it determines that something about the rule (in my case the URL Category) prevents it. It seems like this isn't happening and maybe that's just the way it works, however I find this confusing.

Thanks very much for your insight.

Re: Custom URL Category in security rule - traffic log shows allowed with "any" in URL Cat

herrmoss — Mon, 11 Sep 2017 17:33:23 GMT

So @Remo and @jvalentine, in betwen my other priorities I have been researching this issue with my limited knowledge and understanding.

I have discovered that in 60% of the cases the destination IP Address allowed that shows "any" for URL Category in the logs has been also been allowed sometimes with the correct URL Category showing. I'm not sure why this is and am at a loss to explain the remaining 40% that never show the correct URL Category.

At this point I am willing to accept this behavior but will continue to monitor.

Thanks for your time on this.