GlobalProtect issues after updating firewall version to 10.2.3

[UtkarshKumar]

Hi Team

The customer recently updated one of their firewalls to version 10.2.3 and now when we try to connect to the GlobalProtect client on the end user's machines, we are prompted twice to sign in. The monitoring tab gives a failure with "Authentication failed: empty password". Adding to this, we use Cisco Duo for MFA and we are prompted twice to send a push or enter a passcode every time the client attempts to log in.

The issue only started after upgrading the firewall and there is no issue being experienced on the old firewall version.

The customer has tried to move to the newer GP client version:6.0.3 with no change and also tried reverting back to 6.0.1 and we still have the same issue where the client is prompted twice with Duo Push.

We have verified and recommended the configuration as per Palo Best Practice to Generate and Accept the authentication cookie but still no change.
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u0000004LvbCAE&lang=en_US%E2%80%A...

Device Checks/Custom Checks on the portal are not enabled and thus it is not overriding the Authentication settings.

No other changes have been made to the configuration and the customer stated that the issue was after upgrading to 10.2.3. I do not see any known issues listed and thus would like to confirm if anybody has seen or faced the issue after the upgrade.

I tried checking the logs and can see from authd.log:

Some noticeable logs:
14:50:10.631 -0800 debug: pan_auth_loop(pan_auth_server.c:165): After 300 seconds, authd didn't receive requests, tear down existing socket 14 now
14:51:09.307 -0800 Error: _get_saml_info(pan_authd_saml.c:595): Failed to find cert for in vsys 0
Pan GPS logs shows:

P2727-T19975 12/06/2022 15:38:58:124 Debug(9288): ----Portal Login starts----
P2727-T19975 12/06/2022 15:38:58:124 Debug(2419): Unserialized non-empty cookie for portal lv-gp.korteco.com and user xxxxxx
P2727-T19975 12/06/2022 15:38:58:124 Debug(9310): Cookie exists for saved user xxxxxx. Update saved user to user. Continue for saml
P2727-T19975 12/06/2022 15:38:58:124 Error(9245): GetPassword(): invalid parameter.
P2727-T19975 12/06/2022 15:38:58:124 Debug(14582): Failed to get portal saved password.
P2727-T19975 12/06/2022 15:38:58:124 Debug(11139): Password is empty.
P2727-T19975 12/06/2022 15:38:58:124 Info ( 582): EVP_DecryptFinal_ex failed
P2727-T19975 12/06/2022 15:38:58:124 Debug(9224): Failed to decrypt data
P2727-T19975 12/06/2022 15:38:58:124 Debug(9279): Failed to get portal user password.

P2727-T19975 12/07/2022 06:51:53:507 Debug( 482): error detail is HTTPS User Authentication failure.
P2727-T19975 12/07/2022 06:51:53:507 Debug( 367): received no data
P2727-T19975 12/07/2022 06:51:53:507 Debug( 475): m_bUserAuthentication is set to false.
P2727-T19975 12/07/2022 06:51:53:507 Debug(14333): Auth failed. Private header is auth-failed-password-empty
P2727-T19975 12/07/2022 06:51:53:507 Debug(14362): Auth failed empty password for portal

Detailed Authd.log from the time:
14:45:10.301 -0800 Use "@/tmp/authd.sock", unix domain socket to get authd clients' requests
14:50:10.631 -0800 debug: pan_auth_loop(pan_auth_server.c:165): After 300 seconds, authd didn't receive requests, tear down existing socket 14 now
14:50:10.631 -0800 Use "@/tmp/authd.sock", unix domain socket to get authd clients' requests
14:51:09.304 -0800 debug: pan_auth_request_process(pan_auth_state_engine.c:3612): Receive request: msg type PAN_AUTH_REQ_SAML_CREATE_SSO_REQUEST, conv id 286, body length 2448
14:51:09.304 -0800 debug: _log_saml_input(pan_auth_state_engine.c:2917): Trying to handle SAML/CAS message: <profile: "Duo SSO GlobalProtect", vsys: "vsys1", authd_id: 7172359225543230206 RelayState: "dffe2e79-365f-4d14-b8c3-6820522595ac" 14:51:09.306 -0800 debug: pan_auth_sql_clear_lock_expired_users(pan_authd_sqlite.c:3139): Locklist entries 0, not clearing
14:51:09.307 -0800 Authd in enum phase 4
14:51:09.307 -0800 Error: _get_saml_info(pan_authd_saml.c:595): Failed to find cert for in vsys 0
14:51:09.898 -0800 debug: _retrieve_svr_ids(pan_auth_service.c:645): could not find auth server id vector for Duo SSO GlobalProtect-vsys1-mfa
14:51:09.898 -0800 debug: add_info_from_auth_profile_to_request(pan_auth_util.c:1068): MFA is not configured for the auth profile. No mfa server ids for the user "" (prof/vsys: Duo SSO GlobalProtect/vsys1)
14:51:09.898 -0800 debug: add_info_from_auth_profile_to_request(pan_auth_util.c:1079): MFA configured, but bypassed for GP user ''. (prof/vsys: Duo SSO GlobalProtect/vsys1)
14:51:09.898 -0800 debug: pan_auth_cache_user_is_allowed(pan_auth_cache_allowlist_n_grp.c:571): This is a single vsys platform, group check for allow list is performed on "vsys1"

Any help in this regard would be appreciated.

Thanks.

[UtkarshKumar]

Hi Team

Just wanted to check if anyone has faced this issue. Is there anything we need to check further?

Appreciate any response on this.

[pan219]

is it a pa-220 by any chance? we (at least another person I have direct contact with) has issues since updating to 10.2.3 and GP connections failing. I see the same error messages in gp logs, coming from mobile devices.

https://live.paloaltonetworks.com/t5/general-topics/globalprotect-ios-stuck/td-p/487381

[UtkarshKumar]

Hi

Customer is on PA-820 and this started after upgrading 10.2.3

Authd.logs have reference to:
16:32:38 2022-12-06 16:32:38.616 -0800 debug: _log_saml_input(pan_auth_state_engine.c:2917): Trying to handle SAML/CAS message: <profile: "Duo SSO GlobalProtect", vsys: "vsys1", authd_id: 7172359225543230233 RelayState: "743c3d0d-3f57-48a1-8441-4479fc1567f0" 16:32:38 2022-12-06 16:32:38.616 -0800 Authd in enum phase 4
16:32:38 2022-12-06 16:32:38.616 -0800 Error: _get_saml_info(pan_authd_saml.c:595): Failed to find cert for in vsys 0
16:32:38 2022-12-06 16:32:38.616 -0800 debug: _get_payload(pan_authd_saml_internal.c:1064): b64 decoded payload length=6470.
16:32:38 2022-12-06 16:32:38.617 -0800 Received SAML Assertion from 'https://sso-x.x.x.x.x..sso.duosecurity.com/saml2/sp/DI5UUKRR6P16NSI7NWIP/metadata' from client 'x.x.x.x'
16:32:38 2022-12-06 16:32:38.617 -0800 debug: _extract_sso_attribute(pan_authd_saml_internal.c:526): Got attr name (username) "User.Username" ; value "testuser";
16:32:38 1670373158 ERROR XMLTooling.CredentialResolver.File : unable to stat local resource (/opt/pancfg/mgmt/global/authd/idp.cert)

In globalprotect logs we see: "Authentication failed: empty password"

Can this be the issue do we have re-import the cert?

Thanks

[UtkarshKumar]

The customer has PA-820,

what we see from authd.log file is:

debug: _log_saml_input(pan_auth_state_engine.c:2917): Trying to handle SAML/CAS message: <profile: "Duo SSO GlobalProtect", vsys: "vsys1", authd_id: 7172359225543230232 RelayState: "743c3d0d-3f57-48a1-8441-4479fc1567f0"  16:32:12   2022-12-06 16:32:12.799 -0800 Authd in enum phase 4
 Error:  _get_saml_info(pan_authd_saml.c:595): Failed to find cert for  in vsys 0

debug: pan_auth_cache_user_is_allowed(pan_auth_cache_allowlist_n_grp.c:571): This is a single vsys platform, group check for allow list is performed on "vsys1"

debug: _retrieve_svr_ids(pan_auth_service.c:645): could not find auth server id vector for Duo SSO GlobalProtect-vsys1-mfa

ERROR XMLTooling.CredentialResolver.File :unable to stat local resource (/opt/pancfg/mgmt/global/authd/idp.cert)

Could this be the issue caused by an improper upgrade and if we need to re-import the cert?

[BPry]

@UtkarshKumar,

I would start troubleshooting this by just reimporting the certificate since the logs are saying it can't be found. If that clears things up, then it's a simple enough fix to get things working properly again. 

[EduRamirez]

I am having the same issue...

[UtkarshKumar]

@BPry We tried to reimport the cert but still facing the same issue. The logs shows:

05:31:12.640 -0800 debug: _extract_sso_attribute(pan_authd_saml_internal.c:526): Got attr name (username) "User.Username" ; value "xxxxx";

mp        authd.log                          2022-12-21 05:31:12   1671629472 ERROR XMLTooling.CredentialResolver.File : unable to stat local resource (/opt/pancfg/mgmt/global/authd/idp.cert)

mp        authd.log                          2022-12-21 05:31:12   1671629472 INFO OpenSAML.Utility.SAMLSign : successful signature verification

Client logs:

P1435-T12807 12/21/2022 07:30:14:188 Debug(13923): Portal auth method: saml, auth src: IDP
P1435-T12807 12/21/2022 07:30:14:188 Debug( 339): Original host lv-gp.korteco.com(lv-gp.korteco.com)
P1435-T12807 12/21/2022 07:30:14:188 Debug( 127): set session proxy to 1-0x105b1c2b8.
P1435-T12807 12/21/2022 07:30:14:188 Debug( 561): Portal or gateway login, set connect timeout to 30.0
P1435-T12807 12/21/2022 07:30:14:188 Info ( 419): Timeouts monitor started, LocalDataTask <F5EEE864-38B6-4F93-98F6-EA6A653BBCC0>.<2>, connect timeout 30.0, receive timeout 30.0
P1435-T12807 12/21/2022 07:30:14:398 Info ( 530): Finished with http://x.x.x.x.x.:443
P1435-T12807 12/21/2022 07:30:14:398 Debug( 482): error detail is HTTPS User Authentication failure.
P1435-T12807 12/21/2022 07:30:14:398 Debug( 367): received no data
P1435-T12807 12/21/2022 07:30:14:398 Debug( 475): m_bUserAuthentication is set to false.
P1435-T12807 12/21/2022 07:30:14:398 Debug(14333): Auth failed. Private header is auth-failed-password-empty
P1435-T12807 12/21/2022 07:30:14:398 Debug(14362): Auth failed empty password for portal
P1435-T12807 12/21/2022 07:30:14:398 Debug( 676): GetHttpResponse: m_errorDetails is HTTPS User Authentication failure..

Let us know if we can do anything else to stop Dual Duo push notifications.

[EduRamirez]

You have to downgrade the PAN-OS version that was working before. PAN-OS 10.2.3 is not supporting Duo MFA, which has been confirmed by PANW on a support case, it could be fixed on the 10.2.4 PAN-OS version. 

[UtkarshKumar]

Thanks 

[BENGOOCH]

@UtkarshKumar - did you end up having to re-import the cert to resolve this? Facing a similar issue on PAN-OS 10.1.

[kumaresh_mishra · Accepted solution]

This might be related to PAN-186957 upgrading to 10.2.x from 10.1.6 breaks the IDP configuration. The metadata for authe profile defined with saml idp under global protect drop down doesn't show any value.

[pharney26]

What debug commands did you run? also, what was your solution? We are having the same issue but with PAN-OS 11.0.2, it's really hindering access to our remote users right now. Thank you.

[pkathiriya]

@pharney26  Did you found any workaround for this issue? We are having the same issue with PAN 11.0.2

[pharney26]

As of now, no workaround was found other than remote users re-trying their connection. Support has escalated this issue to tier 2 and i am now waiting on them.