topic Re: Telemetry decided to stop sending in General Topics

Telemetry decided to stop sending

hafenlabs — Fri, 05 Jul 2024 21:34:26 GMT

UPDATE: This was a bug and was fixed after installing 10.2.10

I have seen a couple others post this same issue on Reddit but figured this would be the place to ask. Does anyone know how to kick telemetry back into a working state? It was working and then last month on the 25th was the last time it sent. no errors, no fail messages, no changes, its just not sending.

I have tried restarting the MGMT plane

disable telemetry -> commit -> enable telemetry -> commit

manually generate a telemetry file

no luck. Did PAN force disable telemetry in relation to CVE-2024-3400? I have applied all of the updates/patches/mitigations and validated I have no IOCs so I am not worried about having it enabled specially since it turns out that its not required for the exploit to be successful so at this point I am at a loss as to what to do next aside from submit a TAC case that is really not super important and will just take up space right now while they are dealing with other things.

I was working through BPA assessment evaluations and improvements and now have to fix this to continue 😕

UPDATE:

The CLI will provide more details and so I am now able to see the actual last time it generated a telemetry file was 4/17/24, funny thing is the UI reports the 25th so they don't match. You can see here my attempts to manually kick off the process was today and here is where I learned a bit more... the telemetry generation gets stuck at 80% and sits forever until I issue the command to cancel it.

show device-telemetry collect-now Device Telemetry Collect-Now: Progress in percentage: 0 Latest Request ID: 1715201509 Last Request ID: 1713392833 Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz Last Attempt: Wed May 8 13:51:51 PDT 2024 Last Success: Wed Apr 17 15:27:15 PDT 2024

Kicking it off manually:

request device-telemetry collect-now Successfully initiate device telemetry collect-now. Request ID: 1715208516

Checking the status to see % complete after a few minutes:

show device-telemetry collect-now Device Telemetry Collect-Now: Progress in percentage: 30 Latest Request ID: 1715208516 Last Request ID: 1713392833 Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz Last Attempt: Wed May 8 15:48:38 PDT 2024 Last Success: Wed Apr 17 15:27:15 PDT 2024 admin@PA-440> show device-telemetry collect-now Device Telemetry Collect-Now: Progress in percentage: 40 Latest Request ID: 1715208516 Last Request ID: 1713392833 Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz Last Attempt: Wed May 8 15:48:38 PDT 2024 Last Success: Wed Apr 17 15:27:15 PDT 2024 admin@PA-440> show device-telemetry collect-now Device Telemetry Collect-Now: Progress in percentage: 80 Latest Request ID: 1715208516 Last Request ID: 1713392833 Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz Last Attempt: Wed May 8 15:48:38 PDT 2024 Last Success: Wed Apr 17 15:27:15 PDT 2024

After 5 minutes of checking every so often you can see it gets to 80% and then that is it, it never completes, never finishes, never stops, never ends, unless I tell it to abort the process.

UPDATE2:

nothing useful here in the log file if I tail it

tail lines 200 follow yes mp-log device_telemetry_send.log 2024-05-08 15:13:04,588 dt_send INFO DT - enabled ! 2024-05-08 15:14:04,285 dt_send INFO DT - enabled ! 2024-05-08 15:15:04,285 dt_send INFO DT - enabled ! 2024-05-08 15:16:03,835 dt_send INFO DT - enabled ! 2024-05-08 15:17:04,491 dt_send INFO DT - enabled ! 2024-05-08 15:18:04,202 dt_send INFO DT - enabled ! 2024-05-08 15:19:03,912 dt_send INFO DT - enabled ! 2024-05-08 15:20:04,479 dt_send INFO DT - enabled ! 2024-05-08 15:21:04,068 dt_send INFO DT - enabled ! 2024-05-08 15:22:04,428 dt_send INFO DT - enabled ! 2024-05-08 15:23:04,523 dt_send INFO DT - enabled ! 2024-05-08 15:23:04,537 dt_send INFO check and send: new sending intvl 2024-05-08 15:23:04,539 dt_send INFO send dir all: trying to send files in dir /opt/panlogs/tmp/device_telemetry/hour 2024-05-08 15:23:04,539 dt_send INFO send dir all: trying to send files in dir /opt/panlogs/tmp/device_telemetry/day 2024-05-08 15:24:04,220 dt_send INFO DT - enabled ! 2024-05-08 15:25:03,927 dt_send INFO DT - enabled ! 2024-05-08 15:26:04,602 dt_send INFO DT - enabled ! 2024-05-08 15:27:04,355 dt_send INFO DT - enabled ! 2024-05-08 15:28:04,080 dt_send INFO DT - enabled ! 2024-05-08 15:29:03,798 dt_send INFO DT - enabled ! 2024-05-08 15:30:04,700 dt_send INFO DT - enabled !

Re: Telemetry decided to stop sending

Brandon_Wertz — Wed, 08 May 2024 20:52:05 GMT

Maybe your issue could be solved by this? https://live.paloaltonetworks.com/t5/next-generation-firewall/ngfw-telemetry-uploads-failing/td-p/439841

Re: Telemetry decided to stop sending

hafenlabs — Wed, 08 May 2024 22:42:35 GMT

I looked into that already and it was not related and even after setting it again I am still seeing the same lack of telemetry.

Re: Telemetry decided to stop sending

Steven_E — Sat, 18 May 2024 13:19:11 GMT

Hi @hafenlabs , I have the same issue.
Are you runninh 11.2.0 .
What is your region set to, is it case sensitive? -->

show device-telemetry settings

Can you maybe post the Reddit link?

Thanks

Re: Telemetry decided to stop sending

levi.rogers — Wed, 29 May 2024 03:44:13 GMT

I just noticed this on two of our devices as well. Did you guys ever find anything out?

Re: Telemetry decided to stop sending

hafenlabs — Fri, 31 May 2024 19:16:05 GMT

After this ticket getting passed around TAC and lots of debugging commands run the determination was that this is now a "known issue" and they are "seeing more of these" and the fix will be in upcoming 11.x versions ETA July... I asked about 10.2 and the support rep did not have an ETA for a fix in the 10.2 branch

Re: Telemetry decided to stop sending

hafenlabs — Fri, 31 May 2024 19:17:55 GMT

This is on 10.2.9-hwhateverthelatestisrightnow

seems its now a "known issue" see reply to another user in this thread for more 🙂

Re: Telemetry decided to stop sending

stefan.tomasevic — Thu, 04 Jul 2024 12:35:57 GMT

Same problem. Stuck at 80%.

is there command to abort collecting and start again?

Re: Telemetry decided to stop sending

hafenlabs — Fri, 05 Jul 2024 21:32:34 GMT

There is a command to trigger it to collect now but it will get stuck in the same spot. Mine is working now the fix was included (but not listed in the change log/addressed issues) for 10.2.10

Re: Telemetry decided to stop sending

stefan.tomasevic — Mon, 08 Jul 2024 06:44:24 GMT

There is no fix for this. Simply reload of fw fixed it.

One of mine started working after

debug software restart process reportd

but started working after 12-15h after debug command.

Re: Telemetry decided to stop sending

hafenlabs — Wed, 17 Jul 2024 23:54:57 GMT

That does not and did not work for me and many others that had the issue. PANOS 10.2.10 fixed it for me

Re: Telemetry decided to stop sending

relayer — Tue, 30 Jul 2024 15:08:06 GMT

11.1.2-h3 seems to be affected as well.. Anyone else here running 11.1.2-h3 and happens to know which release this is supposed to be fixed in? Thanks!

Re: Telemetry decided to stop sending

Haniecki — Thu, 01 Aug 2024 08:50:11 GMT

I have version 11.1.2-h3 and there is the same problem.

I also have information that the problem will be solved in version 11.1.5 - ETA 29/08/2024

Re: Telemetry decided to stop sending

KRSuplicki — Fri, 09 Aug 2024 16:03:19 GMT

In your statement, you mention, "Unless I tell it to abort the process", how do you abort the process?

I am currently stuck at 80% nothing I do clears the current Collect-Now status. Assuming I can abort this I may be able to start the process again.

I am on 11.1.2-h3 - regardless I believe this may be a known bug for this release as well as others.

Re: Telemetry decided to stop sending

hafenlabs — Sun, 11 Aug 2024 20:42:35 GMT

You can cancel it with:

request device-telemetry cancel-collect-now

Re: Telemetry decided to stop sending

stefan.tomasevic — Wed, 14 Aug 2024 06:58:24 GMT

This is like a new command. It was not there months ago.

Re: Telemetry decided to stop sending

Goran_ZeroTrust — Wed, 20 Nov 2024 10:57:59 GMT

Telemetry doesn't work, so I've done some troubleshooting. I startet "Generate Telemetry File" 2 days ago. It got stuck at 80%. Found out this was a known bug (PAN-250062), and upgraded to 10.2.12-h2 which should have the fix. Even after a upgrade/restart, the collect now hangs:

Device Telemetry Collect-Now:
Progress in percentage: 80
Latest Request ID: 1731923853
Last Request ID: 0
Last Filename: N/A
Last Attempt: Mon Nov 18 10:57:35 CET 2024
Last Success: N/A

I've tried "request device-telemetry cancel-collect-now", but no change. What to do?

Re: Telemetry decided to stop sending

stefan.tomasevic — Wed, 20 Nov 2024 11:01:48 GMT

The same.

I downgraded to prefered release, and today again to new prefered release and its started working.