WinRM-HTTP fails with the error 401

[Megrretz]

Hello, I'd like to request some advice on trying to shift away from WMI to WinRM-HTTP/S based User-ID. 

I followed the set-up guide by Palo and User-ID server monitoring is able to connect to the domain controller over WinRM-HTTP, but only every hour. If I set session monitoring to something less than 3600 seconds, each attempt by the user-id service to get data from the DC is registered as the following error:

2022-02-21 23:51:04.488 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2736): failed to connect to winrm server http1 in vsys 1
2022-02-21 23:51:04.488 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2780): Connection failed. response code = 401, error: (null) in vsys 1, server=http1.

Then at the hour from the last successful connect it will connect again and get the data from the DC. So the system logs look like this with session monitor set to 20 minutes:

01:35 - Server monitor connected

01:50 - Server monitor connection failed, HTTP code 401, (null)

02:10 - Server monitor connection failed, HTTP code 401, (null)

02:30 - Server monitor connection failed, HTTP code 401, (null)

02:35 - Server monitor connected

etc etc

Does anyone know if there are more settings that need to be set on the DC that are not in the documentation?

Did not see any logs of use on the DC side that would clarify the issue

Thanks!

[BPry]

@Megrretz,

A 401 means that it's not actually authorized, and I would assume that when you're seeing it state it's "connected" it isn't actually passing any data. Should easily be able to see the authentication attempt on the DC side of things in the Security event logs. 

• Double check the DC for any WinRM restrictions that may be in place.
• Check the security event logs on the DC in question; it could also be an issue with the Kerberos ticket, but I would think this unlikely.
• Go back and double check the permissions on the account.

The big thing here is that the security event logs on the DC should be telling you exactly where the authentication failure is happening. You can also perform a WinRM trace with the account and see where the breakdown is. 

[Megrretz]

Hey Bpry, thanks for your reply!

It does authorize but only once each hour, if the probes are set more frequently it's then that I see the 401 errors. These once-an-hour probes do get data, I see it under Monitor -> User-ID -> data source - active-directory. The problem is of course that getting data once an hour is not workable

My guess is that there's some sort of limit how often these WinRM requests can be made but I don't know if it's a Kerberos limitation, WinRM or something else on the domain controller... Since the data is infrequently gathered it seems like there is no issue with the permissions or other restrictions, besides the 1-per-hour.

On the DC event logs under System I checked thoroughly and I do not see any connection failures for the userid service account or any errors or warnings for that matter. Could it be that the logs would be saved to a folder under the Windows logs on the DC and you have to enable them first?

In the Palo CLI:

Server: http1(vsys: vsys1)
Host: xxx
Last error: Authentication failed
num of log query made : 4615
num of log query failed : 4489 <<<<<<<<<<<<<<<<<<<<<<< fails when query more frequent than 1 hour
num of log read : 113876 <<<<<<<<<<<<<<<<<<< logs being read when the query does not fail
last record timestamp : 1645609267
last record time : 20220223094107.0-000

Another line from the logs mentions TGT:

2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2736): failed to connect to winrm server http1 in vsys 1
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2780): Connection failed. response code = 401, error: (null) in vsys 1, server=http1.
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_server_record(pan_user_id_win.c:3568): TGT is changed at 1645609268, Server monitor http1(vsys1): connection failed, HTTP code 401, (null)

I also turned on WMI as a test and it works fine:

Server: WMI test(vsys: vsys1) (job 772321)
Host: xxx
num of log query made : 27
num of log query failed : 0
num of log read : 3852
last record timestamp : 1645609629
last record time : 20220223094709.158917-000

[Megrretz]

I can't see my reply that I sent earlier, trying again

Thank you for your message BPry, I checked the security logs on the domain controller's event logs and there is nothing for the service account used for the HTTP-WinRM user-id and no warnings or errors even though the UserID service is probing each 15 seconds... Is it possible that the logs should be going somewhere else (e.g. the Windows -> Kerberos service folders in the event logs) ?

Data is being passed every hour to the firewall when the service account makes a successful connection and i see it under the monitor -> user-id with the source being active-directory:

Server: http1(vsys: vsys1)
Host: ...
Last error: Authentication failed
num of log query made : 5040
num of log query failed : 4912 
num of log read : 115019 <<<<<<<<<<<<<<<< logs being read every hour
last record timestamp : 1645613535
last record time : 20220223105215.0-000

So what it looks like is that there is some Windows side limitation for WinRM that only allows the logs to be gathered every 60 minutes which is not frequent enough, but I am having problems finding what it could be due to (kerberos, winrm, account time limitation?). From what I gather the 401 http error is because the service account is already logged in or something similar

Performing the following command I also do not see what could be limiting this to 60 minutes:

winrm get winrm/config/

Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000 <<<<<<<<<<<<<<<

[sounetworksupport]

Hi, Megrretz. We are having very nearly the same problem in our environment since we switched to using the PAN-OS integrated User-ID agent via WinRM-HTTP for User-ID mapping. We switched from the WMI method.

The issue only affects one out of our four domain controllers. All our domain controllers are running Windows Server 2012 R2 and have the exact same GPO applied to them as a group. I've validated that the winrm config is identical between the malfunctioning DC and the other DCs, and I don't see anything set in our winrm configuration that would explain the once-per-hour limit. I see the connection succeeding every hour for a minute or two, then it goes back to receiving 401 errors from the DC until another hour has passed.

Sadly, I don't have any useful information to share other than to say that you're not alone. If I figure something out, I'll share what I learn here, and I hope you'll do the same.