Beginning sometime last week (possibly on 12/26) our Windows-based User-ID agent stopped being able to query our DCs for user-to-IP mappings. The PA shows 1000s of request for IP mappings msgs with little to no response msgs from the agent. The agent server debug log shows a long queue of pending lookups with occasional WMI/Netbios access errors. PA support walked us thru a bunch of the standard stuff without coming up with a definite cause. We recreated the config, changed users/passwords, and changed settings multiple times without success.
Today, we were finally able to get the User-ID agent working again by giving the agent Domain Admins permissions. The docs say the agent requires Distributed COM Users and Event Log Readers group permissions and then: "the PAN-OS integrated User-ID agent... requires Server Operator privileges to monitor user sessions." "the Windows-based User-ID agent... does not require Server Operator privileges". As far as we can tell we have never had the User-ID agent account as a member of Server Operators and that group doesn't even seem to exist (Windows docs say it no longer exists in local, only in AD, but we couldn't find it there either).
Does anyone know of a recent update that may have broken the agent permission? I am aware of the Windows update the broke WMI from the PAN-OS User-ID agent, but we can not find anything that would have broken the Windows User-ID agent.
Server Operators is definitely still an AD group, it's in fact Built in to AD by default at domain/Builtin. Possible that your AD administrators aren't looking in that directory?
I'm not aware of anything having changed that would have broken this, and I have a few environments running with the Windows agent that have the latest security patches that I know don't have the Server Operator role applied to them. I'd be looking in AD to make sure that your other permissions are correct, but it's definitely a permissions issue if Domain Admin fixed things. It's also possible that someone did some security hardening on your AD nodes that stripped away permissions that would have been present by default.
Yeah, both our AD admins say it isn't there which has me a bit perplexed and we couldn't add the group to the user. When I do a 'net group /domain' or 'net group "server operators" /domain' it is not there, but 'net group "domain admins" /domain' is there.
I spent all day Thursday and today going back and forth and testing with the AD/server/Infosec admins, everyone claims nothing related has changed. Nothing shown on the Windows agent server or DCs as a recently applied Windows update. It seems like an AD policy or update that stripped permissions, but we can't find it.
There was this issue from about 6 months ago
"Installing Microsoft's June 8th 2021 NTLM Elevation of Privilege Vulnerability patches may break the User-ID Agent's connection to Domain Controller(s)"
There was also this issue from a few months ago
I'm not sure what specific patches this thread refers to, but the comments mostly suggest ditching WMI for WinRM
What are you referring to when you say "Windows update the broke WMI from the PAN-OS User-ID agent" ? I've been keeping an eye on these issues, but my PAN-OS User-ID lookups over WMI still seem to be working.
There are 2 different UserID agents (plus the GP, etc.): the PAN-OS agent that runs on the firewall (Device->User Identification->User Mapping); and the Windows agent that runs as a service on a separate server (Device->User Identification->User-ID agent).
The PAN-OS agent queries the DCs directly from the firewall using WMI or WinRM (v9.0 and later). The Sept update broke our PAN-OS agent WMI queries, we have been running v8.1 so WinRM wasn't available. I have upgraded 1 firewall to v9.1, but we have not gotten WinRM to connect yet (may be a different firewall issue elsewhere).
The Windows UserID agent runs on a separate server and queries the DCs/Exchange servers event logs for login/logout events, and directly probes clients using WMI and Netbios. The PA connects to the server and retrieves the discovered user-IP mappings. I looked at the June 8th patch problem (KB50036xx, multiple patches depending on WinOS), but didn't find any matches on the server or DC. We had to elevate the permissions on the agent service account to be able to query the DC's event logs again. Still looking but don't know why...
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!