NGFW Telemetry Uploads Failing

[KMcKenna]

We have been receiving critical alerts saying telemetry uploads on all of our NGFWs from all locations are failing since just past midnight EDT last night.   The most relevant parts of the alert are:

type: SYSTEM
subtype: device-telemetry
eventid: send-failed
object:
fmt: 0
id: 0
module: general
severity: critical
opaque: Failed to send: file 'PA_<redacted>_dt_10.0.5_20211009_0507_4-hr-interval_HOUR.tgz'

Opened a High Severity support ticket but do not expect a response from Palo Alto until Monday given SLA for High severity Support Tickets.

We are licensed through Later 2022.  Support Active.   Not an ISP issue as this is happening at four separate sites across the USA.

Assumed a Palo Alto maintenance issue but this seems to be going on longer than I would expect for maintenance.  Reporting telemetry isn't a critical function, unless it's indicative of some other issue.

Anyone else seeing these issues?

[BPry]

@KMcKenna,

I think this was actually throughout the United States (at least), and I was personally pretty upset to be woken up due to a critical alert being thrown that is no where near critical in functionality. The fact that the device-telemetry is failing to send isn't a concern, and I wouldn't worry about the warning. 

If you have system log settings enabled that email you critical alerts, you can add 'and not (subtype eq device-telemetry)' to your filter to stop getting alerts for telemetry events. 

[KMcKenna · Accepted solution]

@BPry

Thanks for commenting.   I heard back from Support.   The suggested that the issue is the region as it exists in the config, is causing the error.  Support posted the following information.

Also, you can run the command "show device-telemetry settings" and check what region it is showing, and if the region name is shown all in lower case format, we will need to modify it as it is case-sensitive. In your case, I see it says "americas" all in lower case, therefore we would need to change this setting through CLI.

Please use the below commands to modify the region name:

> configure
#set deviceconfig system device-telemetry region Americas
#commit

So what appeared to resolve this so far on one PA-820 was disabling Telemetry and committing.  Then re-enabling Telemetry, Committing... Then setting the region with a capital A and committing again.  It's not clear that this exact process is needed.  But I had tried setting the regain names as directed while Telemetry was active and failing.  Then committing.   But the uipload continued to fail.  So I waited for another cycle.  Saw the same failure and tried this and the telemetry upload was successful on the PA-820 and it's HA paired device.

I just tried the same process on a VM Series. Waiting for status as I believe this is a 2 or 4 hours cycle.  I'll post results.

As to why this became an issue on Oct 8th, I do not know.  We made no changes.  No commits.  No OS upgrade.  I suspect the change may have been made on Palo Alto's side.  We are running 10.0.5 and planning to update the OS to 10.0.6 next weekend.  Possibly the region names in 10.0.5 and earlier are not compliant.

That's what I know.  I hope it helps.  I realize telemetry function isn't critical.  But as you wrote, the critical alerts are annoying.

Thnx again

[LAYER_8]

Thank you for posting the TAC solution. Got these myself, too, and was waiting to hear on workaround for the homelab. 

[KMcKenna]

Meant to post sooner, but for us, editing the Region in the CLI to have an upper case first letter resolve the upload failures.  I suspect this may be related to the version of PAN-OS we are running (10.0.5) or possibly the platforms but we do have three different Strata platforms in service.

So enabling Telemetry, committing, then editing the Region in the CLI and committing again (waiting for commit and HA sync to complete) resolved the issue for us.

Hope this helps.

[rmcrae]

Some things just don't need to be case sensitive.

[Lance]

Good Day,

I am having the same problem on PANOS 10.2.0 and I have tried to steps above but have had no joy. 

Lance

[SMcGregor]

I'm seeing the same problem on upgrade to 10.2.0 on a PA-220.  Tried installing a new certificate, disabled/commit, enabled/commit.  No joy.

[Cromacatene]

same for me!

[SMcGregor]

Screen grabs of the device Certificate and Telemetry status.  I wonder if the absence of any text after "Certificate status" on the Telemetry screen is relevant?

[lkawamoto]

PA3220 on 10.1.5-h2 and started seeing this issue with uploading telemetry a few hours ago (Jun 21 2022).

Tried to update via CLI the region from "america" to "America" and commit on the ACTIVE - this didn't fix the issue.

But then I did the whole turn OFF telemetry, commit (push to passive), turn ON telemetry, commit (push to passive), use CLI to change region from "america" to "America", commit (push to passive) process, and now I do not see the error come in (should have seen it come in about 9 mins ago as of now.)

[KMcKenna]

I've also started seeing this again today at 1:05PM EDT.  Our field is set to "Americas"  We are running 10.0.5.   We are also running 10.0.5 on one other PA-820 and have Telemetry enabled there and are not seeing this.   Other PAs are running 10.1.5-h2.  No issues as of now (9:05PM EDT.

Also.. the last issue we had was at 7:23 EDT.    It had been an issue about every half hour since 1:00pm today.  But 90 minutes have elapsed and I don't see any new alerts.  Just FYI.    If I learn more will post it.    All FWs have region set to Americas.

[ksalustro]

Having similar issue. PA-450 with 10.2.2-h2. Firewall is managed by Panorama.

Region is blank. Status "Failed", Reason "CDL Receiver Key Empty".

If I go to the firewall locally and make any changes to that section, set the region with CLI or GUI, or disable telemetry, it has no effect after hitting Ok. The Commit button is still grayed out. 

The secondary firewall in the HA pair also has a blank region, but its telemetry works.

There's a bug somewhere.