topic Re: Credential agent crashes LSASS in General Topics

Credential agent crashes LSASS

palousermatthew — Tue, 01 Mar 2022 07:12:17 GMT

Setup a 2016 RODC so I could use the Credential Agent.

As soon as I try starting the agent as system, the server pops a message that I will be force restarted in 1 minute. It non-gracefully reboots in 1 minute. I tried agent v10 and v9. Perms and settings appear fine afaik, and suppressing a/v didn't help. Palo sent me a suggestion to roll back patches before Jan or even before July of last year but that doesn't seem right, plus Jan is the baseline in my template. Has anyone experienced a similar issue and had any luck?

Faulting application name: lsass.exe, version: 10.0.14393.4704, time stamp: 0x615be0cd
Faulting module name: samsrv.dll, version: 10.0.14393.4886, time stamp: 0x61d5262e
Exception code: 0xc0000096
Fault offset: 0x000000000000bac6
Faulting process id: 0x298
Faulting application start time: 0x01d82bfd507c5710
Faulting application path: C:\Windows\system32\lsass.exe
Faulting module path: C:\Windows\SYSTEM32\samsrv.dll
Report Id: bf2a0ead-8af1-4d85-b595-2509ddf94f46
Faulting package full name:
Faulting package-relative application ID:

----------

The process wininit.exe has initiated the restart of computer RODC-3 on behalf of user for the following reason: No title for this reason could be found
Reason Code: 0x50006
Shutdown Type: restart
Comment: The system process 'C:\Windows\system32\lsass.exe' terminated unexpectedly with status code -1073741674. The system will now shut down and restart.

----------------

A critical system process, C:\Windows\system32\lsass.exe, failed with status code c0000096. The machine must now be restarted.

Re: Credential agent crashes LSASS

kiwi — Thu, 03 Mar 2022 09:26:21 GMT

Hi @palousermatthew ,

Sounds like a recent MS update broke the credential detection agent. Any update/workaround from support if they are planning a fix or is rolling back currently the only option ?

Cheers,

-Kiwi.

Re: Credential agent crashes LSASS

dennispci — Fri, 04 Mar 2022 08:47:04 GMT

Hi Matthew, I have exactly the same issue. The RODC is running in Azure with the latest patches (I presume, it is a fresh one).

I have a case running with PaloAlto, hopefully there will be a fix fast....

Re: Credential agent crashes LSASS

NaheedVisram — Mon, 07 Mar 2022 20:18:35 GMT

We are having the same issue as well and had to revert patches to get it running again. Tried both v9 and v10 as wekk

Re: Credential agent crashes LSASS

palousermatthew — Tue, 08 Mar 2022 01:48:38 GMT

Which exact patch did folks have to revert to fix this? Just the Feb CU?

Re: Credential agent crashes LSASS

NaheedVisram — Tue, 08 Mar 2022 17:18:10 GMT

We have had the issue with both the January and February CU updates

Re: Credential agent crashes LSASS

JeffCrowell33 — Wed, 09 Mar 2022 18:24:16 GMT

We are experiencing this same issue. We tried to patch the server in Jan and Feb. This caused the server to go into a loop. Once the User-ID Agent service launches it triggers the server to reboot in 1 min. I disabled the User-ID agent so the server is now stable. I looked at the patch and the logs on the server. The windows patch has an update to the Lsass.exe in it. The error log right after the User-ID Agent is triggered it give the error "A Critical system process, C:\windows\system32\lsass.exe, failed with status code c0000096. The machine much now be restarted." When I rolled back the patches on the RODC server I kept getting a BSOD and had to disable the driver on the boot to get the server to load. Once this was up I had to turn the nic back on and I was allowed to launch the User-ID service. I'm trying the March patch now. I'll update what happens after the March patches are installed.

Re: Credential agent crashes LSASS

NaheedVisram — Wed, 09 Mar 2022 20:46:07 GMT

we had the same issues with the march patches as well

Re: Credential agent crashes LSASS

JeffCrowell33 — Wed, 09 Mar 2022 20:48:09 GMT

Still no good after the March update. If I remove the updates the server give a BSOD on boot but I can get the paloalto UserID agent to run. Apply the updates and the server boots normal but once I turn on the UserID agent the server crashes. Hope Paloalto is looking into this.

Re: Credential agent crashes LSASS

btroudy — Thu, 10 Mar 2022 22:25:16 GMT

We are having the same issue, we also have an open support ticket on this. Only resolution at this point is to remove Jan/Feb/Mar CU.

Re: Credential agent crashes LSASS

palousermatthew — Sat, 12 Mar 2022 06:33:28 GMT

Thanks for all the replies. This saved me from sinking additional hours into troubleshooting.

Re: Credential agent crashes LSASS

brucegarlock — Sat, 19 Mar 2022 16:13:00 GMT

I found this via a link in the paloaltonetworks subreddit, and was subjected to the couple of hours trying to figure out what was going on. Luckily, I knew about the botched MS patches that caused bootloops, so uninstalling the patch tipped me off. I then noticed that the cred agent was not connected properly to the FW, so I disabled the service and the server was stable. As soon as I started it, I got the message that the server was going to reboot in one minute. It doesn't even cleanly shutdown, it acts like issuing a reset in VMware.

The last patches that seem to work are December 2021. Since our users continue to fail phishing test emails, I'll keep the Dec 2021 patches on the RODC for now until we get a fix. I did open a case and recommended others do as well so it receives some priority.

Re: Credential agent crashes LSASS

palousermatthew — Sat, 19 Mar 2022 22:09:48 GMT

Thanks for the reply and sorry I didn't reply faster on that thread. Hope this helps gain visibility.

Re: Credential agent crashes LSASS

NaheedVisram — Tue, 22 Mar 2022 20:02:01 GMT

anyone else hear back from support? Ive just been told theyre investigating....

Re: Credential agent crashes LSASS

brucegarlock — Tue, 22 Mar 2022 20:39:02 GMT

I wish... I put in a S2 ticket last Wed, the 16th. I did not even have my case assigned until the 18th. Way behind on the SLA for a S2. The current workaround is to reduce the security of the RODC, or disable the Cred agent. I hope PAN fixes support. Certainly a good problem to have (many new customers needing support), but not good for those of us paying for premium support, and not receiving it.

Re: Credential agent crashes LSASS

JeffCrowell33 — Wed, 23 Mar 2022 14:44:02 GMT

I added a ticket and they wanted some logs. Still nothing at this time.

Re: Credential agent crashes LSASS

Magnus_App — Thu, 31 Mar 2022 12:24:01 GMT

We have a case as well but nothing so far. There is this KB but no more info really.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000004NJKCA2&lang=en_US%E2%80%A9&refURL=http:%2F%2Fknowledgebase.paloaltonetworks.com%2FKCSArticleDetail

Re: Credential agent crashes LSASS

brucegarlock — Thu, 07 Apr 2022 11:14:51 GMT

I got an update on our case and PAN is saying that this is a MS issue. My questions back on the case to PAN are something like the following. Right now, they are pointing at MS as the cause, but I am not clear on the following:

1. Is this a bug in the patch?

2. Is this a new policy MS is now enforcing on domain controllers that would prevent the User ID Cred agent from working properly?

3. Is MS accepting this as a bug and going to fix it?

Re: Credential agent crashes LSASS

AndreasThanner — Wed, 18 May 2022 07:58:39 GMT

Is there a workaround known to anyone? (Except of uninstall of patches on DCs 🙂 )

Re: Credential agent crashes LSASS

JeffCrowell33 — Mon, 11 Apr 2022 12:22:57 GMT

This is disappointing you are correct. In a change meeting with all our IT staff we discussed this issue and it was 100% we will not remove the patches on the DC to make this feature work. We will wait for a fix or abandon Paloalto all together and find something else that will provide these services. Someone on here reported that Paloalto is blaming it on Microsoft. I reached out to Microsoft about this and their response was "It was an issue with one of their application and it fixed their product." Meaning they are not going to change this patch because it resolved an issue they had with their own product. They don't care about Paloalto. Hopefully Paloalto finds a workaround soon otherwise we will be finding a new product.