GRPC status UNAVAILABLE in intelligent offload

[ElliotM]

Hi All,

Has anyone else come across an issue where a process called 'pan_grpcd' is using upwards of 85% of the CPU on a PA-VM.

The VM is running version 11.0.2 and i can see an error in the system logs - 'GRPC status UNAVAILABLE in intelligent offload'

I have logged this with TAC however they seem to be at a dead end currently.

[kiwi]

Hi @ElliotM ,

You can validate Intelligent Traffic Offload using the following command:

admin@auto-pavm> less mp-log pan_grpcd.log

The expected log output for a successful offload is as follows:

[PD] dec free list 0xe0ff022000 
RS LIB INIT in DP! 
pan_fec_app_init: fec_data 0xe0feef1088, maxentries 120 
[FEC] enc free list 0xe0feef1100, dec free list 0xe0feef10b8 
Creating dp grpc ring buf
Initializing dp grpc ring buf
Mapping flow data memory
Found offload parameters
Heart beat found 1
Established connection to offload device

Are you seeing other logs that could shed some more light ?

Source: https://docs.paloaltonetworks.com/vm-series/10-2/vm-series-deployment/set-up-the-vm-series-firewall-...

Kind regards,

-Kim.

[ElliotM]

Hi,

We don't even have this feature/use it. The main issue we are seeing is the pan_grpcd process is using upward of 90% of the CPU!

[Oliver_Schuler]

We see the same issue in a PoC where we deployed two VM series firewalls in Azure using the official Terraform templates from PANW. The process pan_grpcd is consuming most of the management CPU and commits from Panorama take up to 15 minutes!

It seems there's not much technical documentation available on this feature, there's a bunch of CLI commands containing "grpcd", one of them is "debug dataplane process grpcd off". We tried that, it didn't help.

We just opened up a TAC case.

[Oliver_Schuler]

Results from the TAC case:

- it seems to be related to a known internal bug which should be fixed in one of the following target releases: 11.1.0, 11.0.4, 11.2.0, and 10.2.8 (the root engineer couldn't provide me the exact bugfix id).

- with root access (through a PA engineer) the grpcd process could be stopped, not a solution for us as we're redeploying the firewalls frequently (PoC)

- it might help to remove the "Intelligent Traffic Offload" from the subscription. Deleting the "Intelligent Traffic Offload" license key file itself didn't help, but maybe if we redeploy the firewalls...