URL resolving to unknown while know on brightcloud

Reply
Not applicable

URL resolving to unknown while know on brightcloud

Hi guys,

We're facing a weird problem.

We currently have the unknown URL category set to alert in order to log all users traffic.

We tried to modify that because of some weird traffic categorised as unknown and always visiting russian website.

However, this lead us to be impacted by huge amount of calls and our support personnel could not follow. Why? A big majority of the website visited by the users are reported as unknow by the firewall.

As a test, we connect to brightclous and perform a lookup for every of these website. And they resolve!

As a second test, we connect the cli and run the "test url www.url.com" and it resolves!!!!!!

We call a user, take his PC from remote and make the same test from his internet browser. The firewall says category unknown. While 10seconds ago the same url was resolved to the right category from the CLI.

Is this a bug? Or a feature behavior that we're missing?

Appliance is PAN 2050 with software 5.0.4.

Thanks for your help.

Best,

M S filtering

L4 Transporter

Re: URL resolving to unknown while know on brightcloud

Hi,

Can you check to see if any of the URL Filtering profiles on the firewall have 'Dynamic Url' unchecked?

If so, this could be the cause. If using Dynamic URL lookup(brightcloud-cloud) then it is recommended to enable Dynamic URL on ALL URL Filtering profiles used in security policy.

Once changed, it will be required to clear the DP url cache to trigger the new categorization.

> clear url-cache all


-Stefan

L5 Sessionator

Re: URL resolving to unknown while know on brightcloud

Hi sebbarmo,

Before you clear your cache, can you clarify one thing?  On the devices in question, when you run the command "test url", what exactly is the output?  You should typically see two entries, one for the base db and another for the cloud db.  Do both come back as the same answer, or does the base db show "unknown" while the cloud has the expected category?

Thanks,

Doris

Not applicable

Re: URL resolving to unknown while know on brightcloud

Hi,

Thanks for your answers.

Doris,

I only get an output for the base db (Dynamic db). I odn't get any output in regards with cloud db.

Here is an example:

sebbarmo@XXXXXX(active)> test url www.intaircoat.com

www.intaircoat.com society (Dynamic db)

sebbarmo@XXXXXX(active)> test url www.intaircoat.com

www.intaircoat.com society (Dynamic db)

sebbarmo@XXXXXX(active)> test url www.intaircoat.com

www.intaircoat.com society (Dynamic db)


Stefan,


Indeed, I just checked and we got a category where the "dynamic url filtering" checkbox is unchecked.

So I am gonna apply your solution.


But before to do that I wish I could know if its good or not that I only get the "Dynamic DB" answer when testing an url. Is this normal behavior? Or is the normal behavior that I always have to get two answers? If so, What should I do in order to fix this?


Thanks


M S

L5 Sessionator

Re: URL resolving to unknown while know on brightcloud

Hi sebbarmo,

The "dynamic URL filtering" setting basically dictates whether or not you want your device to query the master database in the cloud for an answer, should there be a miss on the device cache and on-device database.  Given that you did not enable this setting, it makes sense that your "test url" output only returned an answer from the dynamic database.

That said, if you actually use a browser to go to www.intaircoat.com do your logs show category "unknown" while the test url results show category "society"?  There shouldn't be a mismatch there.

--Doris

Not applicable

Re: URL resolving to unknown while know on brightcloud

Hi Dyang,

Thanks for the clarifications.

I've made a test with a website not knowed yet by on-device db and you're all right. It first goes to the master db and queries it. If afterwards I perform the same test back again I get answer from the local db which is the same.

sebbarmo@XXXXXXX(active)> test url www.corima-technologies.com

www.corima-technologies.com business-and-economy (Cloud db)

sebbarmo@XXXXXXX(active)> test url www.corima-technologies.com

www.corima-technologies.com business-and-economy (Dynamic db)

Everything's fine up to here.

However, when I try to surf the website using a web browser, I see "unknown" as the category for www.corima-technologies.com and not business-and-economy as expected(see attached screenshot).

The URL filtering profile where the "dynamic url filtering" is unchecked is not used within any Security Profile.

Is this uncked box the source of the above mismatch?

Thank you.

M S

unknown_Category.jpg

L5 Sessionator

Re: URL resolving to unknown while know on brightcloud

That is correct!  Enable dynamic url filtering in the URL filtering profile, and this should address your problem.  As an FYI, while this is a per profile setting, there is also a global setting if you would like to apply this to all URL filtering profiles on your device.  To do this, use the CLI command, "

set deviceconfig setting url dynamic-url yes"

--Doris

Not applicable

Re: URL resolving to unknown while know on brightcloud

Hi Dyang,

I've made the changes and it seems it is giving better results.

However, I've had a few "not-categorized" URLs. But after a page refresh, the website appears properly.

This has probably to do with time reponse of Brightcloud towards our firewall if I ain't wrong? Is this still accurate for PAN-OS v.5 (5sec for timeout and thus not-cvategorized)? Is this a tunable setting?

I'll test this by blocking the unknown category as of tomorow in production and let you know.

Thank you

M S

L4 Transporter

Re: URL resolving to unknown while know on brightcloud

I have the same problem here

Capture1.PNG

testing the url on the CLI for the first time

Capture1.PNG

testing the url on the CLI the second time

Capture1.PNG

All URL profiles have the dynamic check enabled, additionally I have set deviceconfig setting url dynamic-url yes .

But webbrowsing to that url again after the CLI test categorised it correctly the url monitor shows

Capture1.PNG

Does not make sense to me, to fire up a CLI url test before it gets resolved into a category...

The same strange behavior with other in the first place "unknown" or "not-resolved" url's for example www.myexpertone.com

Capture1.PNG

Capture1.PNG

Capture1.PNG

Capture1.PNG

Anyone else ?

L5 Sessionator

Re: URL resolving to unknown while know on brightcloud

Hi,

I met this issue many time during POC, the main reason in my case is, for the first request to an uncategorize web site, the palo send the request to cloud and wait time for an answer. If the answer is not received during this period, URL is taggued as unknown. For sure the answer will be received a little bit latter. Then either the second request or the test in CLI works well :-)

The best solution should be to increase this time-out throught the command: set deviceconfig setting ctd url-wait-timeout.

Please test and let me know.

V.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!