Telemetry decided to stop sending

[hafenlabs]

UPDATE: This was a bug and was fixed after installing 10.2.10 

I have seen a couple others post this same issue on Reddit but figured this would be the place to ask. Does anyone know how to kick telemetry back into a working state? It was working and then last month on the 25th was the last time it sent. no errors, no fail messages, no changes, its just not sending. 

I have tried restarting the MGMT plane

disable telemetry -> commit -> enable telemetry -> commit

manually generate a telemetry file

no luck. Did PAN force disable telemetry in relation to CVE-2024-3400? I have applied all of the updates/patches/mitigations and validated I have no IOCs so I am not worried about having it enabled specially since it turns out that its not required for the exploit to be successful so at this point I am at a loss as to what to do next aside from submit a TAC case that is really not super important and will just take up space right now while they are dealing with other things. 

I was working through BPA assessment evaluations and improvements and now have to fix this to continue 😕

UPDATE:

The CLI will provide more details and so I am now able to see the actual last time it generated a telemetry file was 4/17/24, funny thing is the UI reports the 25th so they don't match. You can see here my attempts to manually kick off the process was today and here is where I learned a bit more... the telemetry generation gets stuck at 80% and sits forever until I issue the command to cancel it.

show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 0
    Latest Request ID: 1715201509
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 13:51:51 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

 Kicking it off manually:

request device-telemetry collect-now

Successfully initiate device telemetry collect-now.
Request ID: 1715208516

Checking the status to see % complete after a few minutes:

show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 30
    Latest Request ID: 1715208516
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 15:48:38 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

admin@PA-440> show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 40
    Latest Request ID: 1715208516
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 15:48:38 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

admin@PA-440> show device-telemetry collect-now

Device Telemetry Collect-Now:
    Progress in percentage: 80
    Latest Request ID: 1715208516
    Last Request ID: 1713392833
    Last Filename: PA_021201035076_dt_10.2.9-h1_20240417_2228_NOW.tgz
    Last Attempt: Wed May  8 15:48:38 PDT 2024
    Last Success: Wed Apr 17 15:27:15 PDT 2024

After 5 minutes of checking every so often you can see it gets to 80% and then that is it, it never completes, never finishes, never stops, never ends, unless I tell it to abort the process.

UPDATE2:

nothing useful here in the log file if I tail it

tail lines 200 follow yes mp-log device_telemetry_send.log

2024-05-08 15:13:04,588 dt_send INFO DT - enabled !
2024-05-08 15:14:04,285 dt_send INFO DT - enabled !
2024-05-08 15:15:04,285 dt_send INFO DT - enabled !
2024-05-08 15:16:03,835 dt_send INFO DT - enabled !
2024-05-08 15:17:04,491 dt_send INFO DT - enabled !
2024-05-08 15:18:04,202 dt_send INFO DT - enabled !
2024-05-08 15:19:03,912 dt_send INFO DT - enabled !
2024-05-08 15:20:04,479 dt_send INFO DT - enabled !
2024-05-08 15:21:04,068 dt_send INFO DT - enabled !
2024-05-08 15:22:04,428 dt_send INFO DT - enabled !
2024-05-08 15:23:04,523 dt_send INFO DT - enabled !
2024-05-08 15:23:04,537 dt_send INFO check and send: new sending intvl
2024-05-08 15:23:04,539 dt_send INFO send dir all: trying to send files in dir /opt/panlogs/tmp/device_telemetry/hour
2024-05-08 15:23:04,539 dt_send INFO send dir all: trying to send files in dir /opt/panlogs/tmp/device_telemetry/day
2024-05-08 15:24:04,220 dt_send INFO DT - enabled !
2024-05-08 15:25:03,927 dt_send INFO DT - enabled !
2024-05-08 15:26:04,602 dt_send INFO DT - enabled !
2024-05-08 15:27:04,355 dt_send INFO DT - enabled !
2024-05-08 15:28:04,080 dt_send INFO DT - enabled !
2024-05-08 15:29:03,798 dt_send INFO DT - enabled !
2024-05-08 15:30:04,700 dt_send INFO DT - enabled !

[stefan.tomasevic]

This is like a new command. It was not there months ago.