Palo Alto User Agent ID Services Failing intermittently

[UtkarshKumar]

We have 2 User ID agents on 2 different windows servers. Both User-ID agents' services are getting hung up and requiring manual intervention of services being stopped and started after couple of days. The issue has progressively been getting worse as well. This is causing group mapping to intermittently fail. We have upgraded the gateways to the latest version PAN-OS 9.1.15 as well but this is still the same. (User-id agent Version 9.1.1-8 and 9.1.4-104)

Checking the user-id agent logs we see:

[Debug 245]: tid 11604: Wmi connect to server x.x.x.x failed with error 0x800706ba

tid 12880: NetBIOS user enumeration error for x.x.x.x - The network path was not found.

tid 13252: Probing IP 172.28.1.56 with WMI failed.
 tid 13252: NetBIOS user enumeration error for x.x.x.x - The network path was not found.

[Error 618]: st->buf[st->index], sizeof(st->buf[st->index]))() failed

[Error 175]: Failed to compose log msg with 68 logs. error -17

[Error 434]: Failed to compose ip-users msg with 31 add 0 delete. error -17
12/16/22 12:55:49:598[Debug 3693]: Device thread 16 handle msg get:user_ip. bodylen 97 xml 1
12/16/22 12:55:49:598[Debug 439]: Composed ip-users msg with 31 add and 0 delete.
12/16/22 12:55:49:598[Error 190]: SSL write error: 1-10054!

From firewall at the same time from user-id logs we see:

Error: pan_user_id_agent_uia_proc_v5(pan_user_id_uia_v5.c:1254): pan_user_id_agent_send_and_recv_msgs() failed for UCMSFTPDC1(1)

-0600 Error: pan_user_id_agent_send_and_recv_msgs(pan_user_id_agent.c:3177): pan_user_msgs_recv() failed

Error: pan_ssl_writen_ext(pan_ssl_utils.c:1288): SSL :error:00000000:lib(0):func(0):reason(0)

Error: pan_user_msg_buf_sendout(pan_user_msg.c:1097): pan_ssl_writen(295) failed

Does this indicate any memory issues on the user agent side? What can we do to further resolve the issue. We do not want to manually restart the services every time on user-id agent.

Thanks in advance.

[kiwi]

Hi @UtkarshKumar ,

If you haven't done so already then I'd recommend increasing the UID debug level on the agent and FW to get more verbose logging:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClR1CAK

If you are using the UID-agent to parse the AD security event logs, syslog messages or the XML-API to obtain UID-mappings it is strongly recommended to disable WMI probing:

configure-the-windows-based-user-id-agent-for-user-mapping

Hope this helps,

-Kiwi.

[Arnesh]

Hi @UtkarshKumar 

I hope you have checked the basic compatibility matrix for the sanity check.

Can you also let us know if you connection between the UID-Agent and Firewall is in 'connected' status or is that broken?

From the logs, this seems similar to a SSL issue. Have you referred to the KB which explains the SSL connection issue between the UID and firewall?

And as @kiwi mentioned, we should also obtain the verbose level logs to understand the issue further.

Regards,

[UtkarshKumar]

Hi @kiwi // Team

Since, we were seeing "[Error 618]: st->buf[st->index], sizeof(st->buf[st->index]))() failed" -- we increased the WMI services memory and still seeing the services to be stopped intermittently.

We took the Verbose logs as indicated but the only errors we see is:
01/11/23 17:10:32:480[Debug 245]: tid 4660: Wmi connect to server x.x.x.x failed with error 0x80070005
01/11/23 17:10:32:480[Debug 90]: tid 4660: Probing IP x.x.x.x with WMI failed.

The services works fine if we restart the process.

Thus as per the best practice, I would like to know what does disabling WMI probing do for us in regards to User-ID, group mapping and ip-user mapping?

I want to make sure we don't break our User-ID mapping because we have policies that use AD group mapping.

Thanks
Utkarsh Kumar