NGFW Telemetry Uploads Failing

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.
Palo Alto Networks Approved
Palo Alto Networks Approved
Community Expert Verified
Community Expert Verified

NGFW Telemetry Uploads Failing

L2 Linker

We have been receiving critical alerts saying telemetry uploads on all of our NGFWs from all locations are failing since just past midnight EDT last night.   The most relevant parts of the alert are:

 

 

type: SYSTEM
subtype: device-telemetry
eventid: send-failed
object:
fmt: 0
id: 0
module: general
severity: critical
opaque: Failed to send: file 'PA_<redacted>_dt_10.0.5_20211009_0507_4-hr-interval_HOUR.tgz'

 

Opened a High Severity support ticket but do not expect a response from Palo Alto until Monday given SLA for High severity Support Tickets.

We are licensed through Later 2022.  Support Active.   Not an ISP issue as this is happening at four separate sites across the USA.

 

Assumed a Palo Alto maintenance issue but this seems to be going on longer than I would expect for maintenance.  Reporting telemetry isn't a critical function, unless it's indicative of some other issue.

 

Anyone else seeing these issues?

1 accepted solution

Accepted Solutions

L2 Linker

@BPry

Thanks for commenting.   I heard back from Support.   The suggested that the issue is the region as it exists in the config, is causing the error.  Support posted the following information.

 

Also, you can run the command "show device-telemetry settings" and check what region it is showing, and if the region name is shown all in lower case format, we will need to modify it as it is case-sensitive. In your case, I see it says "americas" all in lower case, therefore we would need to change this setting through CLI.

Please use the below commands to modify the region name:


> configure
#set deviceconfig system device-telemetry region Americas
#commit

 

So what appeared to resolve this so far on one PA-820 was disabling Telemetry and committing.  Then re-enabling Telemetry, Committing... Then setting the region with a capital A and committing again.  It's not clear that this exact process is needed.  But I had tried setting the regain names as directed while Telemetry was active and failing.  Then committing.   But the uipload continued to fail.  So I waited for another cycle.  Saw the same failure and tried this and the telemetry upload was successful on the PA-820 and it's HA paired device.

I just tried the same process on a VM Series. Waiting for status as I believe this is a 2 or 4 hours cycle.  I'll post results.

 

As to why this became an issue on Oct 8th, I do not know.  We made no changes.  No commits.  No OS upgrade.  I suspect the change may have been made on Palo Alto's side.  We are running 10.0.5 and planning to update the OS to 10.0.6 next weekend.  Possibly the region names in 10.0.5 and earlier are not compliant.

 

That's what I know.  I hope it helps.  I realize telemetry function isn't critical.  But as you wrote, the critical alerts are annoying.

 

Thnx again

View solution in original post

12 REPLIES 12

Cyber Elite
Cyber Elite

@KMcKenna,

I think this was actually throughout the United States (at least), and I was personally pretty upset to be woken up due to a critical alert being thrown that is no where near critical in functionality. The fact that the device-telemetry is failing to send isn't a concern, and I wouldn't worry about the warning. 

If you have system log settings enabled that email you critical alerts, you can add 'and not (subtype eq device-telemetry)' to your filter to stop getting alerts for telemetry events. 

L2 Linker

@BPry

Thanks for commenting.   I heard back from Support.   The suggested that the issue is the region as it exists in the config, is causing the error.  Support posted the following information.

 

Also, you can run the command "show device-telemetry settings" and check what region it is showing, and if the region name is shown all in lower case format, we will need to modify it as it is case-sensitive. In your case, I see it says "americas" all in lower case, therefore we would need to change this setting through CLI.

Please use the below commands to modify the region name:


> configure
#set deviceconfig system device-telemetry region Americas
#commit

 

So what appeared to resolve this so far on one PA-820 was disabling Telemetry and committing.  Then re-enabling Telemetry, Committing... Then setting the region with a capital A and committing again.  It's not clear that this exact process is needed.  But I had tried setting the regain names as directed while Telemetry was active and failing.  Then committing.   But the uipload continued to fail.  So I waited for another cycle.  Saw the same failure and tried this and the telemetry upload was successful on the PA-820 and it's HA paired device.

I just tried the same process on a VM Series. Waiting for status as I believe this is a 2 or 4 hours cycle.  I'll post results.

 

As to why this became an issue on Oct 8th, I do not know.  We made no changes.  No commits.  No OS upgrade.  I suspect the change may have been made on Palo Alto's side.  We are running 10.0.5 and planning to update the OS to 10.0.6 next weekend.  Possibly the region names in 10.0.5 and earlier are not compliant.

 

That's what I know.  I hope it helps.  I realize telemetry function isn't critical.  But as you wrote, the critical alerts are annoying.

 

Thnx again

Thank you for posting the TAC solution. Got these myself, too, and was waiting to hear on workaround for the homelab. 

Help the community! Add tags and mark solutions please.

L2 Linker

Meant to post sooner, but for us, editing the Region in the CLI to have an upper case first letter resolve the upload failures.  I suspect this may be related to the version of PAN-OS we are running (10.0.5) or possibly the platforms but we do have three different Strata platforms in service.

 

So enabling Telemetry, committing, then editing the Region in the CLI and committing again (waiting for commit and HA sync to complete) resolved the issue for us.

 

Hope this helps.

L3 Networker

Some things just don't need to be case sensitive.

Good Day,

 

I am having the same problem on PANOS 10.2.0 and I have tried to steps above but have had no joy. 

 

Lance

I'm seeing the same problem on upgrade to 10.2.0 on a PA-220.  Tried installing a new certificate, disabled/commit, enabled/commit.  No joy.

same for me!

Screen grabs of the device Certificate and Telemetry status.  I wonder if the absence of any text after "Certificate status" on the Telemetry screen is relevant?

Screenshot (29).pngScreenshot (30).png

L0 Member

PA3220 on 10.1.5-h2 and started seeing this issue with uploading telemetry a few hours ago (Jun 21 2022).

 

Tried to update via CLI the region from "america" to "America" and commit on the ACTIVE - this didn't fix the issue.

 

But then I did the whole turn OFF telemetry, commit (push to passive), turn ON telemetry, commit (push to passive), use CLI to change region from "america" to "America", commit (push to passive) process, and now I do not see the error come in (should have seen it come in about 9 mins ago as of now.)

L2 Linker

I've also started seeing this again today at 1:05PM EDT.  Our field is set to "Americas"  We are running 10.0.5.   We are also running 10.0.5 on one other PA-820 and have Telemetry enabled there and are not seeing this.   Other PAs are running 10.1.5-h2.  No issues as of now (9:05PM EDT.

 

Also.. the last issue we had was at 7:23 EDT.    It had been an issue about every half hour since 1:00pm today.  But 90 minutes have elapsed and I don't see any new alerts.  Just FYI.    If I learn more will post it.    All FWs have region set to Americas.

L3 Networker

Having similar issue. PA-450 with 10.2.2-h2. Firewall is managed by Panorama.

Region is blank. Status "Failed", Reason "CDL Receiver Key Empty".

If I go to the firewall locally and make any changes to that section, set the region with CLI or GUI, or disable telemetry, it has no effect after hitting Ok. The Commit button is still grayed out. 

The secondary firewall in the HA pair also has a blank region, but its telemetry works.

There's a bug somewhere. 

  • 1 accepted solution
  • 21123 Views
  • 12 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!