FAO: PAN. Brightcloud Server Down / Miscategorization of websites based on Dynamic dB on version 3955. Why??

[kalyanram.piratla]

Dear Palo Alto Networks,

Off late, I have been seeing some bizzare issues with Brightcloud services version 3954 and 3955.  As per what I could see under the system logs, on 09/10/12, the brightcloud server could not receive data and there was failure in retrieving data (in-spite of having all the network connections checked and were absolutely fine).  Then on 10/10/12, connection fails to Brightcloud server causing major issues across various locations where PAN has been deployed. 

The issue was many educational institues block the category "not-resolved" and with Brightcloud going down and with no Dynamic dB query (in-spite of having Dynamic URL Filtering under URL Filtering profile checked) did cause havoc.

Please see snapshots below:

Can someone please let me know why all this has happened?  Was any kind of maintenance work taking place?  If yes, did I miss any warnings from PAN (as per my knowledge, I don't think so). 

I know version 3956 is out and the first thing I will be told is to upgrade which I will be doing it or rather done it as I am writing this. 

But I would really appreciate if someone kindly explains the reason behind this issue.

Look forward to hearing from PAN.

Many Thanks

Kind Regards,

Kalyan 

[dyang]

Hi Kalyan,

There was indeed an outage at BrightCloud yesterday - they're currently working with their service providers to identify the root cause of the failure and working on improving their failover procedures to reduce the impact on customers such as yourself.  I'm hopeful that the chances of this type of outage will be minimized in the future and appreciate everyone's patience and understanding as BrightCloud works to resolve their issues.

Thanks,

Doris

[LCMember1607]

They're still down for me.

[LCMember1607]

admin@PAN-Primary(active)> tail follow yes mp-log pan_bc_download.log

Oct 11 14:02:25 Cannot receive data from  'service.brightcloud.com:80' to download BrightCloud URL database

Oct 11 14:02:25 Error downloading latest URL database

Oct 16 07:29:10  ip 208.87.136.156 message RT time 0.046

Oct 16 07:29:10  ip 208.87.136.247 message RT time 0.013

Oct 16 07:29:10  ip 64.87.3.54 message RT time 0.019

Oct 16 07:29:10  ip 94.236.25.159 message RT time 0.020

Oct 16 07:29:10 Best IP for service.brightcloud.com is 208.87.136.247

Oct 16 07:29:10 Connected to Brightcloud update server service.brightcloud.com

Oct 16 07:29:10 Cannot receive data from  'service.brightcloud.com:80' to download BrightCloud URL database

Oct 16 07:29:10 Error downloading latest URL database

[PANUser-1234]

We also have problems with Brightcloud Updates.

Are there any know issues at the moment?

Oct 17 16:57:47  ip 208.87.136.156 message RT time 0.336
Oct 17 16:57:48  ip 208.87.136.247 message RT time 0.329
Oct 17 16:57:48  ip 94.236.25.159 message RT time 0.061
Oct 17 16:57:48  ip 64.87.3.54 message RT time 0.350
Oct 17 16:57:48 Best IP for service.brightcloud.com is 94.236.25.159
Oct 17 16:57:48 Connected to Brightcloud update server service.brightcloud.com, initiated by 172.25.102.2
Oct 17 16:57:49 Newer update available...
Oct 17 16:57:49  ip 23.23.101.87 message RT time 0.531
Oct 17 16:57:50  ip 46.137.242.211 message RT time 1.114
Oct 17 16:57:51  ip 54.247.160.55 message RT time 0.230
Oct 17 16:57:51  ip 94.236.25.159 message RT time 0.163
Oct 17 16:57:51 Best IP for database.brightcloud.com is 94.236.25.159
Oct 17 16:57:51 Connected to Brightcloud update server database.brightcloud.com, initiated by 172.25.102.2
Oct 17 16:57:51 Error: Missing Content-Length:
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Server: Microsoft-HTTPAPI/2.0
Date: Wed, 17 Oct 2012 14:57:50 GMT

Oct 17 16:57:51 Failed to download 'part_bcdb_3.961.bin'
Oct 17 16:57:51 Error: pan_bc_download(pan_bc_url.c:1506): Failed to perform download and update
Oct 17 16:57:51 Error downloading latest URL database

Regards,

Andreas

[LCMember1607]

We recently purchased the URL Filtering package, but have been running a PAN for the last 2 years.

To get it to work I had to set the Service Route Configuration in setup to connect with my public interface.

[kalyanram.piratla]

Well, it will be more useful if the cause behind the outage was disclosed as I have people on my back asking for the reason.

With regards to error in downloading the URL database, this seems to a mixed bag now..

[rowinchester]

I am gettign the same error when trying to downlaod a new URL signature. We are running 4.1.6 and have not been able to update from  3995.:

BrightCloud URL filtering database update initiated

(active)> tail follow yes mp-log pan_bc_download.log
Jan 22 10:54:13 Newer update available...
Jan 22 10:54:15  ip 54.245.232.43 message RT time 0.377
Jan 22 10:54:15  ip 54.243.45.243 message RT time 0.083
Jan 22 10:54:16  ip 46.137.175.75 message RT time 0.444
Jan 22 10:54:17  ip 46.137.204.246 message RT time 1.166
Jan 22 10:54:18  ip 54.248.249.141 message RT time 0.815
Jan 22 10:54:18 Best IP for database.brightcloud.com is 54.243.45.243
Jan 22 10:54:18 Connected to Brightcloud update server database.brightcloud.com, initiated by 192.168.10.14
Jan 22 10:54:18 URL database download: 0% done
Jan 22 10:54:22 URL database download: 1% done

Jan 22 10:54:25 URL database download: 2% done
Jan 22 10:54:29 URL database download: 3% done
Jan 22 10:54:33 URL database download: 4% done
Jan 22 10:54:36 URL database download: 5% done
Jan 22 10:54:39 URL database download: 6% done
Jan 22 10:54:43 URL database download: 7% done
Jan 22 10:54:46 URL database download: 8% done
Jan 22 10:54:49 URL database download: 9% done
Jan 22 10:54:53 URL database download: 10% done
Jan 22 10:54:56 URL database download: 11% done
Jan 22 10:54:59 URL database download: 12% done
Jan 22 10:55:03 URL database download: 13% done
Jan 22 10:55:07 URL database download: 14% done
Jan 22 10:55:10 URL database download: 15% done
Jan 22 10:55:13 URL database download: 16% done
Jan 22 10:55:16 URL database download: 17% done
Jan 22 10:55:20 URL database download: 18% done
Jan 22 10:58:16 Failed to download 'full_bcdb_4.28.bin'
Jan 22 10:58:16 Error: pan_bc_download(pan_bc_url.c:1501): Failed to perform download and update
Jan 22 10:58:16 Error downloading latest URL database
Received header HTTP/1.1 200 OK
Content-Length: 369697034
Server: Microsoft-HTTPAPI/2.0
Date: Tue, 22 Jan 2013 15:53:08 GMT

error in SSL_read() call