URL resolving to unknown while know on brightcloud

[sebbarmo]

Hi guys,

We're facing a weird problem.

We currently have the unknown URL category set to alert in order to log all users traffic.

We tried to modify that because of some weird traffic categorised as unknown and always visiting russian website.

However, this lead us to be impacted by huge amount of calls and our support personnel could not follow. Why? A big majority of the website visited by the users are reported as unknow by the firewall.

As a test, we connect to brightclous and perform a lookup for every of these website. And they resolve!

As a second test, we connect the cli and run the "test url www.url.com" and it resolves!!!!!!

We call a user, take his PC from remote and make the same test from his internet browser. The firewall says category unknown. While 10seconds ago the same url was resolved to the right category from the CLI.

Is this a bug? Or a feature behavior that we're missing?

Appliance is PAN 2050 with software 5.0.4.

Thanks for your help.

Best,

M S filtering

[sspringer]

Hi,

Can you check to see if any of the URL Filtering profiles on the firewall have 'Dynamic Url' unchecked?

If so, this could be the cause. If using Dynamic URL lookup(brightcloud-cloud) then it is recommended to enable Dynamic URL on ALL URL Filtering profiles used in security policy.

Once changed, it will be required to clear the DP url cache to trigger the new categorization.

> clear url-cache all

-Stefan

[dyang]

Hi sebbarmo,

Before you clear your cache, can you clarify one thing?  On the devices in question, when you run the command "test url", what exactly is the output?  You should typically see two entries, one for the base db and another for the cloud db.  Do both come back as the same answer, or does the base db show "unknown" while the cloud has the expected category?

Thanks,

Doris

[sebbarmo]

Hi,

Thanks for your answers.

Doris,

I only get an output for the base db (Dynamic db). I odn't get any output in regards with cloud db.

Here is an example:

sebbarmo@XXXXXX(active)> test url www.intaircoat.com

www.intaircoat.com society (Dynamic db)

sebbarmo@XXXXXX(active)> test url www.intaircoat.com

www.intaircoat.com society (Dynamic db)

sebbarmo@XXXXXX(active)> test url www.intaircoat.com

www.intaircoat.com society (Dynamic db)

Stefan,

Indeed, I just checked and we got a category where the "dynamic url filtering" checkbox is unchecked.

So I am gonna apply your solution.

But before to do that I wish I could know if its good or not that I only get the "Dynamic DB" answer when testing an url. Is this normal behavior? Or is the normal behavior that I always have to get two answers? If so, What should I do in order to fix this?

Thanks

M S

[dyang]

Hi sebbarmo,

The "dynamic URL filtering" setting basically dictates whether or not you want your device to query the master database in the cloud for an answer, should there be a miss on the device cache and on-device database.  Given that you did not enable this setting, it makes sense that your "test url" output only returned an answer from the dynamic database.

That said, if you actually use a browser to go to www.intaircoat.com do your logs show category "unknown" while the test url results show category "society"?  There shouldn't be a mismatch there.

--Doris

[sebbarmo]

Hi Dyang,

Thanks for the clarifications.

I've made a test with a website not knowed yet by on-device db and you're all right. It first goes to the master db and queries it. If afterwards I perform the same test back again I get answer from the local db which is the same.

sebbarmo@XXXXXXX(active)> test url www.corima-technologies.com

www.corima-technologies.com business-and-economy (Cloud db)

sebbarmo@XXXXXXX(active)> test url www.corima-technologies.com

www.corima-technologies.com business-and-economy (Dynamic db)

Everything's fine up to here.

However, when I try to surf the website using a web browser, I see "unknown" as the category for www.corima-technologies.com and not business-and-economy as expected(see attached screenshot).

The URL filtering profile where the "dynamic url filtering" is unchecked is not used within any Security Profile.

Is this uncked box the source of the above mismatch?

Thank you.

M S

[dyang]

That is correct!  Enable dynamic url filtering in the URL filtering profile, and this should address your problem.  As an FYI, while this is a per profile setting, there is also a global setting if you would like to apply this to all URL filtering profiles on your device.  To do this, use the CLI command, "

set deviceconfig setting url dynamic-url yes"

--Doris

[sebbarmo]

Hi Dyang,

I've made the changes and it seems it is giving better results.

However, I've had a few "not-categorized" URLs. But after a page refresh, the website appears properly.

This has probably to do with time reponse of Brightcloud towards our firewall if I ain't wrong? Is this still accurate for PAN-OS v.5 (5sec for timeout and thus not-cvategorized)? Is this a tunable setting?

I'll test this by blocking the unknown category as of tomorow in production and let you know.

Thank you

M S

[gafrol]

I have the same problem here

testing the url on the CLI for the first time

testing the url on the CLI the second time

All URL profiles have the dynamic check enabled, additionally I have set deviceconfig setting url dynamic-url yes .

But webbrowsing to that url again after the CLI test categorised it correctly the url monitor shows

Does not make sense to me, to fire up a CLI url test before it gets resolved into a category...

The same strange behavior with other in the first place "unknown" or "not-resolved" url's for example www.myexpertone.com

Anyone else ?

[VinceM]

Hi,

I met this issue many time during POC, the main reason in my case is, for the first request to an uncategorize web site, the palo send the request to cloud and wait time for an answer. If the answer is not received during this period, URL is taggued as unknown. For sure the answer will be received a little bit latter. Then either the second request or the test in CLI works well 🙂

The best solution should be to increase this time-out throught the command: set deviceconfig setting ctd url-wait-timeout.

Please test and let me know.

V.

[gafrol]

Ok, wait a minute what is the difference between a MP cloud url lookup and a CLI url test in terms of timeout ?

Anyway I increased the timeout from 5 to 10 seconds and cleared the url cache, didn't change anything.

[dyang]

Vincent is correct - by default, we wait 5 seconds for an answer from the cloud.  If we do not receive an answer back within that time, we will categorize the URL as category "not-resolved" and apply policy accordingly.  Assuming that an answer does come back after that 5 second period, the cache will update with the answer, so the next time you visit the site you will get the correct category, which is what you saw.  There are generally two scenarios in which you could get the "not-resolved" category.  Obviously if the BrightCloud servers are down, then you will not get an answer.  The second scenario is if your MP is currently overloaded and the requests are queued up. 

[sebbarmo]

Hi,

Just to give a little  feedback on our situation.

I've put above solutions in production and indeed, it seems better working than before. Meaning that now, not all the website are categorized as unknow but since I issued the "set deviceconfig setting url dynamic-url yes", cleared the cache. Also we experience almost zero "not-resolved" since I issued  the time for request to the cloud service with the "set deviceconfig setting ctd url-wait-timeout" (Thanks Vincent).

We still have an issue that generates many calls for our support desk.  The issue is there are many, many website used for business purpose that are uncategorized and thus resolved to "unknow". Leading to blockage of web traffic towards those website.

It seems that Brightcloud doesn't categorize the websites(or some of them) the way it should(wrong category. Ex: religion[forbidden] while it's governamental agency...) or doesn't categorize them at all !!!!

It also seems that the problem is occuring mainly on european(we're located in Belgium - the need resources in Belgium and europe) websites.

Any news on how to solve this? Is there a known solution besides contacting Brightcloud everyday with a 10 to 20 websites to change category or to categorized?

Thanks.

M S

[dyang]

Hi sebbarmo,

Unfortunately, that is a coverage/accuracy issue with the BrightCloud database itself.  Submitting the change requests to BrightCloud should help, but given that you are submitting multiple requests per day, please note that you can also submit a list of URLs to dbchange@brightcloud.com in lieu of submitting them individually via the website.

If you'd like to have a more in-depth discussion regarding BrightCloud quality issues, please contact your SE so that we can set that up.

Thanks,

Doris

[Rags]

Thank you dyang!