02-22-2022 03:18 AM
Hello, I'd like to request some advice on trying to shift away from WMI to WinRM-HTTP/S based User-ID.
I followed the set-up guide by Palo and User-ID server monitoring is able to connect to the domain controller over WinRM-HTTP, but only every hour. If I set session monitoring to something less than 3600 seconds, each attempt by the user-id service to get data from the DC is registered as the following error:
2022-02-21 23:51:04.488 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2736): failed to connect to winrm server http1 in vsys 1
2022-02-21 23:51:04.488 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2780): Connection failed. response code = 401, error: (null) in vsys 1, server=http1.
Then at the hour from the last successful connect it will connect again and get the data from the DC. So the system logs look like this with session monitor set to 20 minutes:
01:35 - Server monitor connected
01:50 - Server monitor connection failed, HTTP code 401, (null)
02:10 - Server monitor connection failed, HTTP code 401, (null)
02:30 - Server monitor connection failed, HTTP code 401, (null)
02:35 - Server monitor connected
Does anyone know if there are more settings that need to be set on the DC that are not in the documentation?
Did not see any logs of use on the DC side that would clarify the issue
02-22-2022 10:38 AM
A 401 means that it's not actually authorized, and I would assume that when you're seeing it state it's "connected" it isn't actually passing any data. Should easily be able to see the authentication attempt on the DC side of things in the Security event logs.
The big thing here is that the security event logs on the DC should be telling you exactly where the authentication failure is happening. You can also perform a WinRM trace with the account and see where the breakdown is.
02-23-2022 02:10 AM
Hey Bpry, thanks for your reply!
It does authorize but only once each hour, if the probes are set more frequently it's then that I see the 401 errors. These once-an-hour probes do get data, I see it under Monitor -> User-ID -> data source - active-directory. The problem is of course that getting data once an hour is not workable
My guess is that there's some sort of limit how often these WinRM requests can be made but I don't know if it's a Kerberos limitation, WinRM or something else on the domain controller... Since the data is infrequently gathered it seems like there is no issue with the permissions or other restrictions, besides the 1-per-hour.
On the DC event logs under System I checked thoroughly and I do not see any connection failures for the userid service account or any errors or warnings for that matter. Could it be that the logs would be saved to a folder under the Windows logs on the DC and you have to enable them first?
In the Palo CLI:
Server: http1(vsys: vsys1)
Last error: Authentication failed
num of log query made : 4615
num of log query failed : 4489 <<<<<<<<<<<<<<<<<<<<<<< fails when query more frequent than 1 hour
num of log read : 113876 <<<<<<<<<<<<<<<<<<< logs being read when the query does not fail
last record timestamp : 1645609267
last record time : 20220223094107.0-000
Another line from the logs mentions TGT:
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2736): failed to connect to winrm server http1 in vsys 1
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2780): Connection failed. response code = 401, error: (null) in vsys 1, server=http1.
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_server_record(pan_user_id_win.c:3568): TGT is changed at 1645609268, Server monitor http1(vsys1): connection failed, HTTP code 401, (null)
I also turned on WMI as a test and it works fine:
Server: WMI test(vsys: vsys1) (job 772321)
num of log query made : 27
num of log query failed : 0
num of log read : 3852
last record timestamp : 1645609629
last record time : 20220223094709.158917-000
02-23-2022 03:07 AM
I can't see my reply that I sent earlier, trying again
Thank you for your message BPry, I checked the security logs on the domain controller's event logs and there is nothing for the service account used for the HTTP-WinRM user-id and no warnings or errors even though the UserID service is probing each 15 seconds... Is it possible that the logs should be going somewhere else (e.g. the Windows -> Kerberos service folders in the event logs) ?
Data is being passed every hour to the firewall when the service account makes a successful connection and i see it under the monitor -> user-id with the source being active-directory:
Server: http1(vsys: vsys1)
Last error: Authentication failed
num of log query made : 5040
num of log query failed : 4912
num of log read : 115019 <<<<<<<<<<<<<<<< logs being read every hour
last record timestamp : 1645613535
last record time : 20220223105215.0-000
So what it looks like is that there is some Windows side limitation for WinRM that only allows the logs to be gathered every 60 minutes which is not frequent enough, but I am having problems finding what it could be due to (kerberos, winrm, account time limitation?). From what I gather the 401 http error is because the service account is already logged in or something similar
Performing the following command I also do not see what could be limiting this to 60 minutes:
winrm get winrm/config/
AllowRemoteShellAccess = true
IdleTimeout = 7200000 <<<<<<<<<<<<<<<
03-10-2022 11:09 AM
Hi, Megrretz. We are having very nearly the same problem in our environment since we switched to using the PAN-OS integrated User-ID agent via WinRM-HTTP for User-ID mapping. We switched from the WMI method.
The issue only affects one out of our four domain controllers. All our domain controllers are running Windows Server 2012 R2 and have the exact same GPO applied to them as a group. I've validated that the winrm config is identical between the malfunctioning DC and the other DCs, and I don't see anything set in our winrm configuration that would explain the once-per-hour limit. I see the connection succeeding every hour for a minute or two, then it goes back to receiving 401 errors from the DC until another hour has passed.
Sadly, I don't have any useful information to share other than to say that you're not alone. If I figure something out, I'll share what I learn here, and I hope you'll do the same.
04-13-2022 10:52 PM
I experienced the same issue.
It was connected normally after two actions.
# First Action
The firewall's DNS IP settings were set differently than the Kerberos (AD) server IP to work with. -->
The firewall's DNS IP setting has been changed to Kerberos (AD) server IP.
Packets collected by the firewall.
Check the SNameString value of the TGS-REQ packet.
# Second action
You have added permissions to the AD server so that you can use AES 128, AES 256 in the firewall AD account option.
# Firewall PAN-OS: 9.1.12
# AD (kerveros) server: Windows Server 2016
# Link to how to set up Winrm-http
08-10-2022 05:35 AM
Hello How did it end up? Did I also meet this problem
08-10-2022 07:59 AM
No resolution yet. We have narrowed down the problem to Kerberos, but the issue appears to be entirely on the PanOS side. Despite having a service ticket, the firewall fails to include the service ticket in the HTTP auth packet for initiating the WinRM-HTTP session, except for one session per hour when it includes the service ticket and then everything works for that one session.
Palo Alto tech support suggested to us that updating PanOS to 10.1.6-h3 resolved a similar problem for other customers, but we haven't had time to perform the upgrade just yet. Whenever we get a resolution, I will update everyone.
08-10-2022 04:11 PM
I had two users successfully switch from WMI to WinRM-HTTP
But I added two new permissions to my account
Remote Management Users
I have experimented with the demo environment and found winRM-HTTP fails with the error 401
Change it to WinRM -https. However, this solution requires the AD to install a certificate server
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!