NGFW Telemetry Uploads Failing

cancel
Showing results for 
Search instead for 
Did you mean: 

NGFW Telemetry Uploads Failing

L1 Bithead

We have been receiving critical alerts saying telemetry uploads on all of our NGFWs from all locations are failing since just past midnight EDT last night.   The most relevant parts of the alert are:

 

 

type: SYSTEM
subtype: device-telemetry
eventid: send-failed
object:
fmt: 0
id: 0
module: general
severity: critical
opaque: Failed to send: file 'PA_<redacted>_dt_10.0.5_20211009_0507_4-hr-interval_HOUR.tgz'

 

Opened a High Severity support ticket but do not expect a response from Palo Alto until Monday given SLA for High severity Support Tickets.

We are licensed through Later 2022.  Support Active.   Not an ISP issue as this is happening at four separate sites across the USA.

 

Assumed a Palo Alto maintenance issue but this seems to be going on longer than I would expect for maintenance.  Reporting telemetry isn't a critical function, unless it's indicative of some other issue.

 

Anyone else seeing these issues?

1 ACCEPTED SOLUTION

Accepted Solutions

L1 Bithead

@BPry

Thanks for commenting.   I heard back from Support.   The suggested that the issue is the region as it exists in the config, is causing the error.  Support posted the following information.

 

Also, you can run the command "show device-telemetry settings" and check what region it is showing, and if the region name is shown all in lower case format, we will need to modify it as it is case-sensitive. In your case, I see it says "americas" all in lower case, therefore we would need to change this setting through CLI.

Please use the below commands to modify the region name:


> configure
#set deviceconfig system device-telemetry region Americas
#commit

 

So what appeared to resolve this so far on one PA-820 was disabling Telemetry and committing.  Then re-enabling Telemetry, Committing... Then setting the region with a capital A and committing again.  It's not clear that this exact process is needed.  But I had tried setting the regain names as directed while Telemetry was active and failing.  Then committing.   But the uipload continued to fail.  So I waited for another cycle.  Saw the same failure and tried this and the telemetry upload was successful on the PA-820 and it's HA paired device.

I just tried the same process on a VM Series. Waiting for status as I believe this is a 2 or 4 hours cycle.  I'll post results.

 

As to why this became an issue on Oct 8th, I do not know.  We made no changes.  No commits.  No OS upgrade.  I suspect the change may have been made on Palo Alto's side.  We are running 10.0.5 and planning to update the OS to 10.0.6 next weekend.  Possibly the region names in 10.0.5 and earlier are not compliant.

 

That's what I know.  I hope it helps.  I realize telemetry function isn't critical.  But as you wrote, the critical alerts are annoying.

 

Thnx again

View solution in original post

4 REPLIES 4

Cyber Elite
Cyber Elite

@KMcKenna,

I think this was actually throughout the United States (at least), and I was personally pretty upset to be woken up due to a critical alert being thrown that is no where near critical in functionality. The fact that the device-telemetry is failing to send isn't a concern, and I wouldn't worry about the warning. 

If you have system log settings enabled that email you critical alerts, you can add 'and not (subtype eq device-telemetry)' to your filter to stop getting alerts for telemetry events. 

L1 Bithead

@BPry

Thanks for commenting.   I heard back from Support.   The suggested that the issue is the region as it exists in the config, is causing the error.  Support posted the following information.

 

Also, you can run the command "show device-telemetry settings" and check what region it is showing, and if the region name is shown all in lower case format, we will need to modify it as it is case-sensitive. In your case, I see it says "americas" all in lower case, therefore we would need to change this setting through CLI.

Please use the below commands to modify the region name:


> configure
#set deviceconfig system device-telemetry region Americas
#commit

 

So what appeared to resolve this so far on one PA-820 was disabling Telemetry and committing.  Then re-enabling Telemetry, Committing... Then setting the region with a capital A and committing again.  It's not clear that this exact process is needed.  But I had tried setting the regain names as directed while Telemetry was active and failing.  Then committing.   But the uipload continued to fail.  So I waited for another cycle.  Saw the same failure and tried this and the telemetry upload was successful on the PA-820 and it's HA paired device.

I just tried the same process on a VM Series. Waiting for status as I believe this is a 2 or 4 hours cycle.  I'll post results.

 

As to why this became an issue on Oct 8th, I do not know.  We made no changes.  No commits.  No OS upgrade.  I suspect the change may have been made on Palo Alto's side.  We are running 10.0.5 and planning to update the OS to 10.0.6 next weekend.  Possibly the region names in 10.0.5 and earlier are not compliant.

 

That's what I know.  I hope it helps.  I realize telemetry function isn't critical.  But as you wrote, the critical alerts are annoying.

 

Thnx again

View solution in original post

Thank you for posting the TAC solution. Got these myself, too, and was waiting to hear on workaround for the homelab. 

Help the community! Add tags & mark solutions please.

L1 Bithead

Meant to post sooner, but for us, editing the Region in the CLI to have an upper case first letter resolve the upload failures.  I suspect this may be related to the version of PAN-OS we are running (10.0.5) or possibly the platforms but we do have three different Strata platforms in service.

 

So enabling Telemetry, committing, then editing the Region in the CLI and committing again (waiting for commit and HA sync to complete) resolved the issue for us.

 

Hope this helps.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!