WinRM-HTTP fails with the error 401

[Megrretz]

Hello, I'd like to request some advice on trying to shift away from WMI to WinRM-HTTP/S based User-ID. 

I followed the set-up guide by Palo and User-ID server monitoring is able to connect to the domain controller over WinRM-HTTP, but only every hour. If I set session monitoring to something less than 3600 seconds, each attempt by the user-id service to get data from the DC is registered as the following error:

2022-02-21 23:51:04.488 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2736): failed to connect to winrm server http1 in vsys 1
2022-02-21 23:51:04.488 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2780): Connection failed. response code = 401, error: (null) in vsys 1, server=http1.

Then at the hour from the last successful connect it will connect again and get the data from the DC. So the system logs look like this with session monitor set to 20 minutes:

01:35 - Server monitor connected

01:50 - Server monitor connection failed, HTTP code 401, (null)

02:10 - Server monitor connection failed, HTTP code 401, (null)

02:30 - Server monitor connection failed, HTTP code 401, (null)

02:35 - Server monitor connected

etc etc

Does anyone know if there are more settings that need to be set on the DC that are not in the documentation?

Did not see any logs of use on the DC side that would clarify the issue

Thanks!

[BPry]

@Megrretz,

A 401 means that it's not actually authorized, and I would assume that when you're seeing it state it's "connected" it isn't actually passing any data. Should easily be able to see the authentication attempt on the DC side of things in the Security event logs. 

• Double check the DC for any WinRM restrictions that may be in place.
• Check the security event logs on the DC in question; it could also be an issue with the Kerberos ticket, but I would think this unlikely.
• Go back and double check the permissions on the account.

The big thing here is that the security event logs on the DC should be telling you exactly where the authentication failure is happening. You can also perform a WinRM trace with the account and see where the breakdown is. 

[Megrretz]

Hey Bpry, thanks for your reply!

It does authorize but only once each hour, if the probes are set more frequently it's then that I see the 401 errors. These once-an-hour probes do get data, I see it under Monitor -> User-ID -> data source - active-directory. The problem is of course that getting data once an hour is not workable

My guess is that there's some sort of limit how often these WinRM requests can be made but I don't know if it's a Kerberos limitation, WinRM or something else on the domain controller... Since the data is infrequently gathered it seems like there is no issue with the permissions or other restrictions, besides the 1-per-hour.

On the DC event logs under System I checked thoroughly and I do not see any connection failures for the userid service account or any errors or warnings for that matter. Could it be that the logs would be saved to a folder under the Windows logs on the DC and you have to enable them first?

In the Palo CLI:

Server: http1(vsys: vsys1)
Host: xxx
Last error: Authentication failed
num of log query made : 4615
num of log query failed : 4489 <<<<<<<<<<<<<<<<<<<<<<< fails when query more frequent than 1 hour
num of log read : 113876 <<<<<<<<<<<<<<<<<<< logs being read when the query does not fail
last record timestamp : 1645609267
last record time : 20220223094107.0-000

Another line from the logs mentions TGT:

2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2736): failed to connect to winrm server http1 in vsys 1
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_query(pan_user_id_win.c:2780): Connection failed. response code = 401, error: (null) in vsys 1, server=http1.
2022-02-23 04:43:51.558 -0500 Error: pan_user_id_winrm_server_record(pan_user_id_win.c:3568): TGT is changed at 1645609268, Server monitor http1(vsys1): connection failed, HTTP code 401, (null)

I also turned on WMI as a test and it works fine:

Server: WMI test(vsys: vsys1) (job 772321)
Host: xxx
num of log query made : 27
num of log query failed : 0
num of log read : 3852
last record timestamp : 1645609629
last record time : 20220223094709.158917-000

[Megrretz]

I can't see my reply that I sent earlier, trying again

Thank you for your message BPry, I checked the security logs on the domain controller's event logs and there is nothing for the service account used for the HTTP-WinRM user-id and no warnings or errors even though the UserID service is probing each 15 seconds... Is it possible that the logs should be going somewhere else (e.g. the Windows -> Kerberos service folders in the event logs) ?

Data is being passed every hour to the firewall when the service account makes a successful connection and i see it under the monitor -> user-id with the source being active-directory:

Server: http1(vsys: vsys1)
Host: ...
Last error: Authentication failed
num of log query made : 5040
num of log query failed : 4912 
num of log read : 115019 <<<<<<<<<<<<<<<< logs being read every hour
last record timestamp : 1645613535
last record time : 20220223105215.0-000

So what it looks like is that there is some Windows side limitation for WinRM that only allows the logs to be gathered every 60 minutes which is not frequent enough, but I am having problems finding what it could be due to (kerberos, winrm, account time limitation?). From what I gather the 401 http error is because the service account is already logged in or something similar

Performing the following command I also do not see what could be limiting this to 60 minutes:

winrm get winrm/config/

Winrs
AllowRemoteShellAccess = true
IdleTimeout = 7200000 <<<<<<<<<<<<<<<

[sounetworksupport]

Hi, Megrretz. We are having very nearly the same problem in our environment since we switched to using the PAN-OS integrated User-ID agent via WinRM-HTTP for User-ID mapping. We switched from the WMI method.

The issue only affects one out of our four domain controllers. All our domain controllers are running Windows Server 2012 R2 and have the exact same GPO applied to them as a group. I've validated that the winrm config is identical between the malfunctioning DC and the other DCs, and I don't see anything set in our winrm configuration that would explain the once-per-hour limit. I see the connection succeeding every hour for a minute or two, then it goes back to receiving 401 errors from the DC until another hour has passed.

Sadly, I don't have any useful information to share other than to say that you're not alone. If I figure something out, I'll share what I learn here, and I hope you'll do the same.

[Parkkapyong]

I experienced the same issue.

It was connected normally after two actions.

# First Action

The firewall's DNS IP settings were set differently than the Kerberos (AD) server IP to work with. -->

The firewall's DNS IP setting has been changed to Kerberos (AD) server IP.

Packets collected by the firewall.
Check the SNameString value of the TGS-REQ packet.

# Second action

You have added permissions to the AD server so that you can use AES 128, AES 256 in the firewall AD account option.

# Firewall PAN-OS: 9.1.12
# AD (kerveros) server: Windows Server 2016

# Link to how to set up Winrm-http

https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/user-id/map-ip-addresses-to-users/configur...

[Zhiyang_Lv]

Hello   How did it end up? Did I also meet this problem

[sounetworksupport]

No resolution yet. We have narrowed down the problem to Kerberos, but the issue appears to be entirely on the PanOS side. Despite having a service ticket, the firewall fails to include the service ticket in the HTTP auth packet for initiating the WinRM-HTTP session, except for one session per hour when it includes the service ticket and then everything works for that one session.

Palo Alto tech support suggested to us that updating PanOS to 10.1.6-h3 resolved a similar problem for other customers, but we haven't had time to perform the upgrade just yet. Whenever we get a resolution, I will update everyone.

[Zhiyang_Lv]

Hello

I had two users successfully switch from WMI to WinRM-HTTP
But I added two new permissions to my account
Remote Management Users
WinRMRemoteWMIUsers__ grou

I have experimented with the demo environment and found winRM-HTTP fails with the error 401
Change it to WinRM -https. However, this solution requires the AD to install a certificate server

[KrunalLathiya]

The error message you're encountering, specifically "Error: pan_user_id_winrm_query: Connection failed. response code = 401," indicates an authentication issue when attempting to connect to your domain controller over WinRM.

1. Check WinRM Configuration on the Domain Controller

• WinRM Service: Ensure that the WinRM service is running on the domain controller. You can check this with the command winrm quickconfig in a PowerShell or Command Prompt window on the domain controller.

• Authentication Method: Verify that the authentication method set up on the domain controller for WinRM matches what the Palo Alto User-ID expects. Typically, this would be Basic Authentication or Kerberos.

• HTTP vs HTTPS: If you're using WinRM-HTTP, consider whether it's necessary to switch to HTTPS, which is more secure. If using HTTPS, ensure that a valid certificate is installed and configured on the domain controller for WinRM.

2. Review Palo Alto User-ID Configuration

• Credential Verification: Double-check the credentials used in the User-ID configuration to ensure they have the necessary permissions on the domain controller.

• Session Interval: While it's unusual that connections only succeed every hour, this might be tied to a configuration setting or a policy on the network or the domain controller. Verify that there isn't a policy that limits the frequency of connections.