BrightCloud URL-Filter is rubbish

[mhuels]

We just have declared to block the category "spam-urls" on the firewall.

Since then, a lot of normal german URLs is blocked wrongly. For instance www.ske.de (banking) or www.paderborn.com (official city-website).

I am wondering, how a website could be a email spam category. Or do i think wrong?

mfg

Manfred

[rferry]

Have you tried submitting the URL and/or domains to Brightcloud to have the research the sites and potentially re-categorize?  The also ask what you believe the site should be classifed as.

I do wonder how long the process takes and also wonder why the URL pattern database is only retrieved once a day (unlike the AV and Threats that are updated once an hour).

BF

[mhuels]

Yes i send some hints to Brightcloud. But after a while i removed the block rule because of the large number of Websites in this category. I guess, the category is for Email-Application-Rules only and not very reliably.

[dyang]

Hello,

BrightCloud classifies URLs as "spam-URL" as URLs that are contained in spam messages.  You can find the full list of category descriptions here:

http://www.brightcloud.com/support/catdescription.php

Please do continue to submit category change requests as they will only help to improve the database.  Should you also have a long list of URLs that you would like to submit changes for, you may also open a ticket with support to submit your entire list.

Thanks,

Doris

[dagibbs]

rferry wrote:
Have you tried submitting the URL and/or domains to Brightcloud to have the research the sites and potentially re-categorize?  The also ask what you believe the site should be classifed as.
I do wonder how long the process takes and also wonder why the URL pattern database is only retrieved once a day (unlike the AV and Threats that are updated once an hour).
BF

I've had to reclassify numerous URL's, and generally the process is completed in 48 hours (24 for them to modify the database and publish, another 2 for it to be applied to the PA in the next scheduled download).

The URL database is only retrieved once a day because as far as I can tell Brightcloud only publish updates once a day - so there's no point in doing it more often.

Cheers.

[migration]

Hi,

One thing I have noticed with the URL classification is if you dont have the dynamic update box ticked some URL's are misclassified as they are not in the initial cache that it downloads.

This is just my experience, I may be wrong.

Marc

[jhickey]

Every URL Filtering product has false positives. I support Websense for many years and it was a regular occurance. That being said, I would hope that if the database is consistently wrong that PA would look to fund a different one. That's my 2 cents.

Justin

[mharding]

We have a big problem with small churches having their websites hosted on the site servers that adult websites, and malware sites are hosted on. I just keep submitting them to Brightcloud.

I think that's the big reason Palo Alto included custom URL categories in PAN-OS 3.1.

[HITSSEC]

We have also noticed porn sites having a default web page that says "This is my Church".  We submit the URL to Brightcloud and we use the custom URL categories to add the sites as work-allowed until Brightcloud updates their database. We sometime use the lookup services of other URL filtering services to double check the site. We also use custom url categories for IP based urls that we identify and our DMZ hosts that are accessed using local DNS entries and thus are not FQDNs which result in being categorised as unknown.

Hope this helps.

Phil

[dagibbs]

HITSSEC wrote:
We have also noticed porn sites having a default web page that says "This is my Church".  We submit the URL to Brightcloud and we use the custom URL categories to add the sites as work-allowed until Brightcloud updates their database. We sometime use the lookup services of other URL filtering services to double check the site. We also use custom url categories for IP based urls that we identify and our DMZ hosts that are accessed using local DNS entries and thus are not FQDNs which result in being categorised as unknown.
Hope this helps.
Phil 

You don't need to go to the extent of adding a custom URL category to make this work.

There's an option in the URL filtering profile setup labelled "allow list" - if you're 100% sure of the site concerned, or if Brightcloud won't re-classify the URL into a category you think it should be in, just put the URL into this field (sans the http:// or https:// protocol extension) and it'll be allowed through regardless of the brightcloud category it falls into.

I use this one as a quick-fix while I wait for brightcloud to re-classify the URL if the user who needs it is really complaining and doesn't want to wait 48 hours.

Cheers.