topic Re: GRPC status UNAVAILABLE in intelligent offload in General Topics

GRPC status UNAVAILABLE in intelligent offload

ElliotM — Wed, 09 Aug 2023 15:34:19 GMT

Hi All,

Has anyone else come across an issue where a process called 'pan_grpcd' is using upwards of 85% of the CPU on a PA-VM.

The VM is running version 11.0.2 and i can see an error in the system logs - 'GRPC status UNAVAILABLE in intelligent offload'

I have logged this with TAC however they seem to be at a dead end currently.

Re: GRPC status UNAVAILABLE in intelligent offload

kiwi — Thu, 10 Aug 2023 09:46:51 GMT

Hi @ElliotM ,

You can validate Intelligent Traffic Offload using the following command:

admin@auto-pavm> less mp-log pan_grpcd.log

The expected log output for a successful offload is as follows:

[PD] dec free list 0xe0ff022000 RS LIB INIT in DP! pan_fec_app_init: fec_data 0xe0feef1088, maxentries 120 [FEC] enc free list 0xe0feef1100, dec free list 0xe0feef10b8 Creating dp grpc ring buf Initializing dp grpc ring buf Mapping flow data memory Found offload parameters Heart beat found 1 Established connection to offload device

Are you seeing other logs that could shed some more light ?

Source: https://docs.paloaltonetworks.com/vm-series/10-2/vm-series-deployment/set-up-the-vm-series-firewall-on-kvm/intelligent-traffic-offload

Kind regards,

-Kim.

Re: GRPC status UNAVAILABLE in intelligent offload

ElliotM — Tue, 15 Aug 2023 14:43:43 GMT

Hi,

We don't even have this feature/use it. The main issue we are seeing is the pan_grpcd process is using upward of 90% of the CPU!

Re: GRPC status UNAVAILABLE in intelligent offload

Oliver_Schuler — Fri, 03 Nov 2023 09:57:07 GMT

We see the same issue in a PoC where we deployed two VM series firewalls in Azure using the official Terraform templates from PANW. The process pan_grpcd is consuming most of the management CPU and commits from Panorama take up to 15 minutes!

It seems there's not much technical documentation available on this feature, there's a bunch of CLI commands containing "grpcd", one of them is "debug dataplane process grpcd off". We tried that, it didn't help.

We just opened up a TAC case.

Re: GRPC status UNAVAILABLE in intelligent offload

Oliver_Schuler — Wed, 22 Nov 2023 10:47:59 GMT

Results from the TAC case:

- it seems to be related to a known internal bug which should be fixed in one of the following target releases: 11.1.0, 11.0.4, 11.2.0, and 10.2.8 (the root engineer couldn't provide me the exact bugfix id).

- with root access (through a PA engineer) the grpcd process could be stopped, not a solution for us as we're redeploying the firewalls frequently (PoC)

- it might help to remove the "Intelligent Traffic Offload" from the subscription. Deleting the "Intelligent Traffic Offload" license key file itself didn't help, but maybe if we redeploy the firewalls...