Telemetry decided to stop sending

[hafenlabs]

UPDATE: This was a bug and was fixed after installing 10.2.10 

I have seen a couple others post this same issue on Reddit but figured this would be the place to ask. Does anyone know how to kick telemetry back into a working state? It was working and then last month on the 25th was the last time it sent. no errors, no fail messages, no changes, its just not sending. 

I have tried restarting the MGMT plane

disable telemetry -> commit -> enable telemetry -> commit

manually generate a telemetry file

no luck. Did PAN force disable telemetry in relation to CVE-2024-3400? I have applied all of the updates/patches/mitigations and validated I have no IOCs so I am not worried about having it enabled specially since it turns out that its not required for the exploit to be successful so at this point I am at a loss as to what to do next aside from submit a TAC case that is really not super important and will just take up space right now while they are dealing with other things. 

I was working through BPA assessment evaluations and improvements and now have to fix this to continue 😕

UPDATE:

The CLI will provide more details and so I am now able to see the actual last time it generated a telemetry file was 4/17/24, funny thing is the UI reports the 25th so they don't match. You can see here my attempts to manually kick off the process was today and here is where I learned a bit more... the telemetry generation gets stuck at 80% and sits forever until I issue the command to cancel it.

show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 0
    Latest Request ID: 1715201509
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 13:51:51 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

 Kicking it off manually:

request device-telemetry collect-now

Successfully initiate device telemetry collect-now.
Request ID: 1715208516

Checking the status to see % complete after a few minutes:

show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 30
    Latest Request ID: 1715208516
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 15:48:38 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

admin@PA-440> show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 40
    Latest Request ID: 1715208516
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 15:48:38 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

admin@PA-440> show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 80
    Latest Request ID: 1715208516
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 15:48:38 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

After 5 minutes of checking every so often you can see it gets to 80% and then that is it, it never completes, never finishes, never stops, never ends, unless I tell it to abort the process.

UPDATE2:

nothing useful here in the log file if I tail it

tail lines 200 follow yes mp-log device_telemetry_send.log

2024-05-08 15:13:04,588 dt_send INFO DT - enabled !
2024-05-08 15:14:04,285 dt_send INFO DT - enabled !
2024-05-08 15:15:04,285 dt_send INFO DT - enabled !
2024-05-08 15:16:03,835 dt_send INFO DT - enabled !
2024-05-08 15:17:04,491 dt_send INFO DT - enabled !
2024-05-08 15:18:04,202 dt_send INFO DT - enabled !
2024-05-08 15:19:03,912 dt_send INFO DT - enabled !
2024-05-08 15:20:04,479 dt_send INFO DT - enabled !
2024-05-08 15:21:04,068 dt_send INFO DT - enabled !
2024-05-08 15:22:04,428 dt_send INFO DT - enabled !
2024-05-08 15:23:04,523 dt_send INFO DT - enabled !
2024-05-08 15:23:04,537 dt_send INFO check and send: new sending intvl
2024-05-08 15:23:04,539 dt_send INFO send dir all: trying to send files in dir /opt/panlogs/tmp/device_telemetry/hour
2024-05-08 15:23:04,539 dt_send INFO send dir all: trying to send files in dir /opt/panlogs/tmp/device_telemetry/day
2024-05-08 15:24:04,220 dt_send INFO DT - enabled !
2024-05-08 15:25:03,927 dt_send INFO DT - enabled !
2024-05-08 15:26:04,602 dt_send INFO DT - enabled !
2024-05-08 15:27:04,355 dt_send INFO DT - enabled !
2024-05-08 15:28:04,080 dt_send INFO DT - enabled !
2024-05-08 15:29:03,798 dt_send INFO DT - enabled !
2024-05-08 15:30:04,700 dt_send INFO DT - enabled !

[Brandon_Wertz]

Maybe your issue could be solved by this?  https://live.paloaltonetworks.com/t5/next-generation-firewall/ngfw-telemetry-uploads-failing/td-p/43... 

[hafenlabs]

I looked into that already and it was not related and even after setting it again I am still seeing the same lack of telemetry.

[Steven_E]

Hi @hafenlabs , I have the same issue.
Are you runninh 11.2.0 .
What is your region set to, is it case sensitive? --> 

show device-telemetry settings

Can you maybe post the Reddit link?

Thanks

[levi.rogers]

I just noticed this on two of our devices as well.  Did you guys ever find anything out?

[hafenlabs]

After this ticket getting passed around TAC and lots of debugging commands run the determination was that this is now a "known issue" and they are "seeing more of these" and the fix will be in upcoming 11.x versions ETA July... I asked about 10.2 and the support rep did not have an ETA for a fix in the 10.2 branch

[hafenlabs]

This is on 10.2.9-hwhateverthelatestisrightnow

seems its now a "known issue" see reply to another user in this thread for more 🙂

[stefan.tomasevic]

Same problem. Stuck at 80%.

is there command to abort collecting and start again?

[hafenlabs]

There is a command to trigger it to collect now but it will get stuck in the same spot. Mine is working now the fix was included (but not listed in the change log/addressed issues) for 10.2.10

[stefan.tomasevic]

There is no fix for this. Simply reload of fw fixed it.

One of mine started working after 

debug software restart process reportd

but started working after 12-15h after debug command.

[hafenlabs]

That does not and did not work for me and many others that had the issue. PANOS 10.2.10 fixed it for me

[relayer]

11.1.2-h3 seems to be affected as well.. Anyone else here running 11.1.2-h3 and happens to know which release this is supposed to be fixed in? Thanks!

[Haniecki]

I have version 11.1.2-h3 and there is the same problem. 

I also have information that the problem will be solved in version 11.1.5 - ETA 29/08/2024

[KRSuplicki]

In your statement, you mention, "Unless I tell it to abort the process", how do you abort the process?

I am currently stuck at 80% nothing I do clears the current Collect-Now status. Assuming I can abort this I may be able to start the process again. 

I am on 11.1.2-h3 - regardless I believe this may be a known bug for this release as well as others.

[hafenlabs]

You can cancel it with:

request device-telemetry cancel-collect-now