10-31-2019 05:48 AM - edited 12-02-2021 06:45 AM
An update for anyone that looks this up later. This is now solved.
We have a mixed environment that is very restrictive, including SSL intercept. In the system logs, we were seeing "CURL ERROR: SSL certificate problem: self signed certificate in certificate chain". In the CLI we were seeing the following with the "tail follow yes mp-log devsrv.log" command:
2019-10-29 22:42:03.319 +0000 Warning: pan_cloud_agent_get_curl_connection(pan_cloud_agent_connect.c:2609): cannot elect a cloud
2019-10-29 22:42:03.319 +0000 curl error: SSL certificate problem: self signed certificate in certificate chain
2019-10-29 22:42:03.320 +0000 Failed to open connection with the cloud after 12360 consecutive tries.
2019-10-29 22:42:04.420 +0000 CLOUD_ELECTION: in wait_t 0 secs.
2019-10-29 22:42:04.544 +0000 Error: verify_cb(pan_ssl_curl_utils.c:614): Error with certificate at depth: 2
2019-10-29 22:42:04.544 +0000 Error: verify_cb(pan_ssl_curl_utils.c:616): Basic Validation of x509 cert Fail ; Code : 19
...
2019-10-29 22:42:04.544 +0000 Error: verify_cb(pan_ssl_curl_utils.c:625): Failed to validate x509 cert from ctx: (19) self signed certificate in certificate chain
2019-10-29 22:42:04.544 +0000 Warning: pan_cloud_agent_collect_cloud_info_cb(pan_cloud_agent_connect.c:1957): cloud elect connection close
I found the available docs were not complete for the changes between 8.x and 9.x PanDB process.
In the end, I found that 8.x and 9.x are using different destinations for updates and since 9.x does not use a seed file, it has to reach the destination to get everything going.
Here are the destinations that are being used for both:
8.x PanDB - 65.154.226.123 "dl1prod.urlcloud.paloaltonetworks.com"
9.x PanDB - 65.154.226.124 "pandb2qa.urlcloud.paloaltonetworks.com" See message from Ldemos below for the proper URL.
In the end, it was the SSL intercept that was keeping the access to pandb2qa.urlcloud.paloaltonetworks.com from working. Once SSL intercept was removed everything started working within the next 10 minutes.
While you can turn off the "verify update server identity" for updates, this does not appear to be an option with the PanDB access, the certificate chain is verified and will not work with an SSL intercept in the middle.
Hopefully I put in enough key words for this to be found if someone else is running into this issue.
