Device certificate is not renewing automatically

[ben.sellick]

Hi all, hoping someone may be able to assist with an issue.

We are seeing that every 3 months our PA device certificate is expiring which causes issues fetching updates from various cloud services (URL filtering, wildfire, update server etc).

Upon renewing the device certificate manually using the OTP in the CSP, the process works and the new certificate is installed fine. It seems just that the automatic renewal process is not working?

I can see the below logs within a tech-support dump that indicate the firewall is aware of the expiring cert, and attempts to renew it +15 days from expiry:

Device_Certgen.log

2022-12-28 04:28:36,218 device_certgen INFO Renewing device certificate
2022-12-28 04:28:37,400 device_certgen INFO Secret_key generated
2022-12-28 04:28:37,400 device_certgen INFO Generated pkey and CSR
2022-12-29 04:33:52,635 device_certgen INFO Renewing device certificate
2022-12-29 04:33:53,493 device_certgen INFO Secret_key generated
2022-12-29 04:33:53,494 device_certgen INFO Generated pkey and CSR
2022-12-30 04:41:24,267 device_certgen INFO Renewing device certificate
2022-12-30 04:41:26,385 device_certgen INFO Secret_key generated
2022-12-30 04:41:26,385 device_certgen INFO Generated pkey and CSR
2022-12-31 04:09:24,314 device_certgen INFO Renewing device certificate
2022-12-31 04:09:26,013 device_certgen INFO Secret_key generated
2022-12-31 04:09:26,013 device_certgen INFO Generated pkey and CSR
2023-01-01 04:42:07,632 device_certgen INFO Renewing device certificate
2023-01-01 04:42:10,039 device_certgen INFO Secret_key generated
2023-01-01 04:42:10,039 device_certgen INFO Generated pkey and CSR
2023-01-02 04:46:43,610 device_certgen INFO Renewing device certificate
2023-01-02 04:46:45,492 device_certgen INFO Secret_key generated
2023-01-02 04:46:45,492 device_certgen INFO Generated pkey and CSR
2023-01-03 04:22:00,381 device_certgen INFO Renewing device certificate
2023-01-03 04:22:01,157 device_certgen INFO Secret_key generated
2023-01-03 04:22:01,157 device_certgen INFO Generated pkey and CSR
2023-01-04 04:40:39,431 device_certgen INFO Renewing device certificate
2023-01-04 04:40:40,627 device_certgen INFO Secret_key generated
2023-01-04 04:40:40,627 device_certgen INFO Generated pkey and CSR
2023-01-05 04:55:41,253 device_certgen INFO Renewing device certificate
2023-01-05 04:55:43,810 device_certgen INFO Secret_key generated
2023-01-05 04:55:43,810 device_certgen INFO Generated pkey and CSR
2023-01-06 04:28:23,482 device_certgen INFO Renewing device certificate
2023-01-06 04:28:25,111 device_certgen INFO Secret_key generated
2023-01-06 04:28:25,111 device_certgen INFO Generated pkey and CSR
2023-01-07 04:30:09,274 device_certgen INFO Renewing device certificate
2023-01-07 04:30:11,694 device_certgen INFO Secret_key generated
2023-01-07 04:30:11,694 device_certgen INFO Generated pkey and CSR
2023-01-08 04:21:50,503 device_certgen INFO Renewing device certificate
2023-01-08 04:21:52,227 device_certgen INFO Secret_key generated
2023-01-08 04:21:52,227 device_certgen INFO Generated pkey and CSR
2023-01-09 04:17:50,467 device_certgen INFO Renewing device certificate
2023-01-09 04:17:52,349 device_certgen INFO Secret_key generated
2023-01-09 04:17:52,349 device_certgen INFO Generated pkey and CSR
2023-01-10 04:27:53,587 device_certgen INFO Renewing device certificate
2023-01-10 04:27:55,029 device_certgen INFO Secret_key generated
2023-01-10 04:27:55,029 device_certgen INFO Generated pkey and CSR
2023-01-11 04:50:00,334 device_certgen INFO Renewing device certificate
2023-01-11 04:50:01,125 device_certgen INFO Secret_key generated
2023-01-11 04:50:01,125 device_certgen INFO Generated pkey and CSR
2023-01-11 16:20:34,528 device_certgen ERROR Device certificate has expired
2023-01-11 16:20:34,565 device_certgen INFO Removing device certificate
2023-01-11 16:20:34,565 device_certgen INFO Removing device certificate
2023-01-11 16:20:34,577 device_certgen INFO Deleting certificates in /opt/pancfg/mgmt/ssl/private (PID: 6020)!
2023-01-11 16:20:34,577 device_certgen INFO Deleting certificates in /opt/pancfg/mgmt/ssl/private (PID: 6020)!
2023-01-11 16:21:37,212 device_certgen INFO Device certificate not found

authd.log

2023-01-11 04:50:03.055 +1100 debug: _device_cert_cb(pan_authd_cas.c:564): change: notify obj 'cfg.device-cert-status', e.g. thermite cert is installed/renewed: was-timestamp xxxxxx ; is-timestamp xxxxxxx
2023-01-11 04:50:03.057 +1100 Device cert (thermite) is renewed, update it in CAS context
2023-01-11 04:50:03.057 +1100 debug: pan_cryptod_sysd_access_key(pan_cryptod_sysd_api.c:1028): Sending GET: id:device_cert_public_key, flag:4 to cryptod
2023-01-11 04:50:03.062 +1100 debug: pan_cryptod_sysd_access_key(pan_cryptod_sysd_api.c:1037): Send GET msg to cryptod for id:device_cert_public_key successful
2023-01-11 04:50:03.062 +1100 debug: pan_cryptod_sysd_access_key(pan_cryptod_sysd_api.c:1057): Received (for id:device_cert_public_key), key data (len=7930):
2023-01-11 04:50:03.062 +1100 debug: pan_cryptod_dump_buf(pan_cryptod_sysd_api.c:749):   [xxx] ...
2023-01-11 04:50:03.062 +1100 debug: pan_cryptod_sysd_access_key(pan_cryptod_sysd_api.c:1028): Sending GET: id:device_cert_private_key, flag:4 to cryptod
2023-01-11 04:50:03.064 +1100 debug: pan_cryptod_sysd_access_key(pan_cryptod_sysd_api.c:1037): Send GET msg to cryptod for id:device_cert_private_key successful
2023-01-11 04:50:03.064 +1100 debug: pan_cryptod_sysd_access_key(pan_cryptod_sysd_api.c:1057): Received (for id:device_cert_private_key), key data (len=3271):
2023-01-11 04:50:03.064 +1100 debug: pan_cryptod_dump_buf(pan_cryptod_sysd_api.c:749):   [xxx] ...
2023-01-11 04:50:03.085 +1100 debug: _populate_device_cert(pan_authd_cas.c:520): device cert expiry epoch = xxxxxxxxxxx
2023-01-11 04:50:03.085 +1100 debug: _populate_device_cert(pan_authd_cas.c:527): device cert subject = /CN=xxxxxxxxxx/O=Palo Alto Networks/L=Santa Clara/ST=CA/C=US/xxxxxxxxx=tpm/xxxxxx=panos/xxxxxxx=xxxxxxxxxx

The system log does not show any events at the time of attempted renewal.

Communication between the device and PAN-DB cloud services is working normally while the device certificate exists, so I do not believe there is a connectivity/communication problem - however is there a specific log file or URL I can test against?

Any advice on troubleshooting further is greatly appreciated.

Thanks.

[kbe]

Our PA failed last night with renew.

Seems like certificatetrusted.paloaltonetworks.com is not reachable at the moment.

So i do not try the manual method which revokes the actual cert. I will wait another day and see what happens this night.

Last time i head the problem that the renew did not work is a while ago on my older 3020 with 9.1.x running. I only had to manually renew it for one time and after that it did the autorenew.

The PA with PanOS 10.2.3 that i have running now is new an running since November, so this would be the first renew now.

But i will not try the manual renew until certificatetrusted.paloaltonetworks.com/ is reachable again.

I also lowered the MTU according to this article (but i did it this morning so it was on MTU 1500 last night when it failed).
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000004NlxCAE