Ok, wait a minute what is the difference between a MP cloud url lookup and a CLI url test in terms of timeout ?
Anyway I increased the timeout from 5 to 10 seconds and cleared the url cache, didn't change anything.
Vincent is correct - by default, we wait 5 seconds for an answer from the cloud. If we do not receive an answer back within that time, we will categorize the URL as category "not-resolved" and apply policy accordingly. Assuming that an answer does come back after that 5 second period, the cache will update with the answer, so the next time you visit the site you will get the correct category, which is what you saw. There are generally two scenarios in which you could get the "not-resolved" category. Obviously if the BrightCloud servers are down, then you will not get an answer. The second scenario is if your MP is currently overloaded and the requests are queued up.
Just to give a little feedback on our situation.
I've put above solutions in production and indeed, it seems better working than before. Meaning that now, not all the website are categorized as unknow but since I issued the "set deviceconfig setting url dynamic-url yes", cleared the cache. Also we experience almost zero "not-resolved" since I issued the time for request to the cloud service with the "set deviceconfig setting ctd url-wait-timeout" (Thanks Vincent).
We still have an issue that generates many calls for our support desk. The issue is there are many, many website used for business purpose that are uncategorized and thus resolved to "unknow". Leading to blockage of web traffic towards those website.
It seems that Brightcloud doesn't categorize the websites(or some of them) the way it should(wrong category. Ex: religion[forbidden] while it's governamental agency...) or doesn't categorize them at all !!!!
It also seems that the problem is occuring mainly on european(we're located in Belgium - the need resources in Belgium and europe) websites.
Any news on how to solve this? Is there a known solution besides contacting Brightcloud everyday with a 10 to 20 websites to change category or to categorized?
Unfortunately, that is a coverage/accuracy issue with the BrightCloud database itself. Submitting the change requests to BrightCloud should help, but given that you are submitting multiple requests per day, please note that you can also submit a list of URLs to email@example.com in lieu of submitting them individually via the website.
If you'd like to have a more in-depth discussion regarding BrightCloud quality issues, please contact your SE so that we can set that up.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!
The Live Community thanks you for your participation!