Issues with Device Telemetry

[southardd]

I have an HA pair (active/passive) of PA3250s (no Panorama) and just recently upgraded to PanOS 10.0.6 from 9.1.9. I configured the device telemetry and downloaded the new certificates for both firewalls. Telemetry is working great on my primary firewall, however, the secondary is failing every time with the error code "CDL Receiver Key Empty". I have opened a support case but have not had much luck. We verified the location is set to "Americas" instead of "americas" and all of the other settings seem to match the working firewall. Has anyone else seen this before? I've attached a screenshot and the text from the email that is generated below.

domain: 1
receive_time: 2021/08/18 08:52:02
seqno: 1608892
actionflags: 0x0
type: SYSTEM
subtype: device-telemetry
config_ver: 0
time_generated: 2021/08/18 08:52:02
high_res_timestamp: 2021-08-18T08:52:02.000-04:00
dg_hier_level_1: 0
dg_hier_level_2: 0
dg_hier_level_3: 0
dg_hier_level_4: 0
vsys_name:
vsys_id: 0
vsys:
eventid: send-failed
object:
fmt: 0
id: 0
module: general
severity: critical
opaque: Failed to send: file 'PA_016301001973_dt_10.0.6_20210817_1607_4-hr-interval_HOUR.tgz'.
dg_id: 0
tpl_id: 0

[BPry]

So simple things first @southardd, do you use the MGMT interface for services or have you setup a service route. The firewall will attempt to send telemetry data out even when you've specified a service route that wouldn't function on your passive firewall. 

[southardd]

Thanks for the reply and sorry for my late response, but both firewalls are set to use the MGMT interface for everything. The config matches on both firewalls so I am not sure why one is working and not the other.

[Matt_Thomas]

I had the same issue on my passive firewall, active firewall does not have issues. Come to find out I had SSL Decryption policies that was preventing the traffic since the CA that Palo Alto is using for 'apitrusted.paloaltonetworks.com' is not a trusted CA (weird). I wasn't decrypting the traffic, just validating certificates. I downloaded the CA cert and imported and marked as "trusted ca". Things seem to be working now. 

LOG: 

show log decryption dst in 35.238.43.180

2021/10/28 04:44:20 ssl Trust 48978 [INTERNAL-IP]
[RULE-NAME] allow Untrust 443 35.238.43.180
TLS1.2 ECDHE AES_256_GCM SHA384 No Decrypt
apitrusted.paloaltonetworks.com Palo Alto Networks Inc.-SJC-Ser
Untrusted issuer CA

[Ben-Price]

@southardd Did you ever get a resolution for this issue?

[southardd]

I did not get an official resolution from Palo Alto, it just kind of stopped on its own and never came back... Not sure what caused it or why it stopped.