2 Step RADIUS Auth Reliability

[apackard]

We are trying to get 2FA RADIUS based authentication working with our Palo's and are seeing unreliable results. After much hunting and teeth gnashing we think we may have found the issue, but not the cause. 

The below is a debug dump from the RADIUS server (RSA AccessID) using Panorama GUI logon as the test case.  The first shows a good 2FA auth and the second a failing one; it looks as if the Palo is not correctly replacing the username attribute in the 2FA RADIUS request (assuming it's not a processing issue on the RADIUS server) so the request is being rejected.

Would anyone know anything about this?  We're particularily confused as it is not consistent - if it always failed I could chalk it up to a bug but it sometimes works and sometimes doesn't...

Will be opening a TAC case but always like to get others users experiences in first!

===========
GOOD AUTH
===========

First factor authentication (LDAP AD username\password) is successful

2018-04-06/17:30:15.568/UTC [AuditEntryProcessor] INFO  RADIUSAUDIT[81] -
----------START_RADIUS_USER_LDAP_AUTHENTICATION----------
EVENTID=RADIUS_USER_LDAP_AUTHENTICATION
DATETIME=Fri Apr 06 17:30:15 UTC 2018
IN_RESPONSE_TO=ed53938a-4193-4a8e-83fe-901b39d3f4ef
DESCRIPTION=RADIUS – LDAP authentication succeeded.
NAS-IP-ADDRESS=<snip>
USER_NAME=jbloggs
CLIENT_ID=RADIUS: Panorama
RADIUS_RESPONSE_TYPE=Access-Challenge
STATUS=SUCCESS
REQUEST_ID=ed53938a-4193-4a8e-83fe-901b39d3f4ef
POLICY_ID=Global Protect Employee VPN
TENANT_ID=<snip>
----------END_RADIUS_USER_LDAP_AUTHENTICATION----------

Second factor authentication (mobile push) is successful - and the username has been replaced with the 2FA method

2018-04-06/17:30:27.973/UTC [AuditEntryProcessor] INFO  RADIUSAUDIT[81] -
----------START_RADIUS_USER_APPROVE_AUTHENTICATION----------
EVENTID=RADIUS_USER_APPROVE_AUTHENTICATION
DATETIME=Fri Apr 06 17:30:27 UTC 2018
IN_RESPONSE_TO=f6f3c7df-0ae1-4c36-bb62-0e4df22db374
DESCRIPTION=RADIUS – Approve method authentication succeeded.
NAS-IP-ADDRESS=<snip>
USER_NAME=MOBILE_METHOD_REFERENCE_ID
CLIENT_ID=RADIUS: Panorama
RADIUS_RESPONSE_TYPE=Access-Accept
STATUS=SUCCESS
REQUEST_ID=f6f3c7df-0ae1-4c36-bb62-0e4df22db374
POLICY_ID=Global Protect Employee VPN
TENANT_ID=<snip>
----------END_RADIUS_USER_APPROVE_AUTHENTICATION----------

=========
BAD AUTH
=========

First factor authentication (LDAP AD username\password) is successful

2018-04-06/17:30:59.015/UTC [AuditEntryProcessor] INFO  RADIUSAUDIT[81] -
----------START_RADIUS_USER_LDAP_AUTHENTICATION----------
EVENTID=RADIUS_USER_LDAP_AUTHENTICATION
DATETIME=Fri Apr 06 17:30:59 UTC 2018
IN_RESPONSE_TO=3e86e627-7498-4663-8fb1-59c6a00707da
DESCRIPTION=RADIUS – LDAP authentication succeeded.
NAS-IP-ADDRESS=<snip>
USER_NAME=jbloggs
CLIENT_ID=RADIUS: Panorama
RADIUS_RESPONSE_TYPE=Access-Challenge
STATUS=SUCCESS
REQUEST_ID=3e86e627-7498-4663-8fb1-59c6a00707da
POLICY_ID=Global Protect Employee VPN
TENANT_ID=<snip>
----------END_RADIUS_USER_LDAP_AUTHENTICATION----------

Second factor authentication fails - and the username has not been replaced.

2018-04-06/17:31:04.017/UTC [AuditEntryProcessor] INFO  RADIUSAUDIT[81] -
----------START_RADIUS_USER_APPROVE_AUTHENTICATION----------
EVENTID=RADIUS_USER_APPROVE_AUTHENTICATION
DATETIME=Fri Apr 06 17:31:04 UTC 2018
IN_RESPONSE_TO=cfd1f7ae-13d4-494a-9d4d-6cd1584f4475
DESCRIPTION=RADIUS – Approve method authentication failed.
NAS-IP-ADDRESS=<snip>
USER_NAME=jbloggs
CLIENT_ID=RADIUS: Panorama
RADIUS_RESPONSE_TYPE=Access-Reject
STATUS=FAIL
REQUEST_ID=cfd1f7ae-13d4-494a-9d4d-6cd1584f4475
POLICY_ID=Global Protect Employee VPN
TENANT_ID=<snip>
----------END_RADIUS_USER_APPROVE_AUTHENTICATION----------

[OtakarKlier]

Hello,

I would look to see how the second radius server is looking the username format. I have seen it as domain\username before and it has caught me out. Also make sure that the radius traffic is allowed by firewall policies.

Regards,

[apackard]

Thanks for the reply.

I've also seen the format sensitivty (it is especially annoying as a failed transparent auth presents a pop-up to enter username\password and it seems to automatically populate the username in domain\name format so we're going to have to educate users to delete the domain portion!).

To be concise however, the second RADIUS auth request is going to the same server (no load balancing etc).

We've done some packet captures of good (left hand) and failing (right hand) sequences.  In both cases everything looks good in terms of expected flows (request -> challenge -> request -> result, challenge 'state token' presented at step 2 and re-presented at step 3, MFA option passed in the password field of step 3) but when it fails it does so in <1ms. 

This is too short (as far as we can tell) for it to try and check the '1' as a password.  We're leaning more towards the RSA RADIUS server having the issue as that seems to be where the actual error is triggered, but cannot discount there is something off with the request that causes it.

[Remo]

Hi @apackard

It actually looked more like a RADIUS server issue to me since the first post in this topic, mainly because the LDAP verification works in both cases and the "error" then happens somehow with the second factor.

What you could do for further troubleshooting is try to reproduce the issue with a tool like this one here: https://www.novell.com/coolsolutions/tools/14377.html

(Already old but still good)

Maybe this way you are able to find the thing that leads to the access-reject on your radius.

Regards,

Remo

[apackard · Accepted solution]

Update to close this one down.

It was an issue with the 2FA provider - the intermediate RADIUS server was randomly rejecting the 2nd factor auth before it had even been attempted!