GP users are getting denied random times

[GeorgiosFakis]

I have global protect v5.0.5 deployed to all Corporate Windows and some users reported that when they work everything stop to work and suddenly after 5-10 minutes is back again without disconnecting them from the global protect .This happen random times and not always .I have a user though that he reports that every day for the last week .

Palo Alto version is 8.1.11 VM-300 and GP agent 5.0.5 on a Windows 10.

I can see from the logs user is working fine in one server , then traffic getting blocked and can see only traffic log but not threat etc. 

I am allowing based on the IP and the Zone and destination is any app any service .

HIP looks fine and agent is sending the report every hour .

Today this happened to a user connected 7 am and stopped working around 10am for 5 minutes .In the logs I can see HIP reports were send before and after the incident and user-id was reported that was learned from the AD .

I can see from the logs if that is helping that user is not written and after working is written . Is that related to USER-ID where I need to exclude the IP pools from the GP on the USER-ID ?

[GeorgiosFakis]

Palo Alto engineer and myself we were looking the logs .

User connected in the morning , opened a UDP session with significant amount of data transimtted and recevied .Was allowed by an ACL in line 35 let's say and after 3 hours Deny ALL acl was matching in line 50 .

We see that HIP report was sent and there flags 0x63 & 0x61 on the allowed and deny from the log .We suspect that is related to HIP report .We see that was sent every hour and HIP log is matching the HIP profile every hour .Question is why traffic that elapsed time was 3 hours is mathcing after that time DENY ALL ACL. 

[MP18]

Thanks for updating on this.

[GeorgiosFakis]

Forgot to update 

We fixed that with disabling the timeout of the user-id but we also upgrade the agents to 5.0.8.

[ksauer507]

I know this is an old thread but I'm experiencing the same issue, 3220 fw's with 10.0.8-h8 and GP 5.2.10-6 on Windows 11.  First I thought it was an expiring cookie, but your right its right after the hip check.  Strange I see exactly the same thing.

So when you say you disabled the timeout of user-id, do you mean you unchecked Enable User Identification Timeout in the cache section in Device > User Identification?

[GeorgiosFakis]

Hi ,

Yes that it is but keep in mind your logging will not be correct because the IP mapping will stay in the firewall for a long time. 

This will might cause issues in your logs for other USER-ID for IP mapping for other zones like server zones where username will stay there. 

For example , if you have a server let's say that is doing some service and has a service account and you RDP , the logs and the IP mapping will have for everything your username unless someone else RDP the server. 

[ksauer507]

Well we tried this and still having problems.

In traffic monitor we just get random denies falling back down to our last rule we call block others (like a catch all deny rule).

In this state seems the user can ping and trace but can't really "access" anything.  Like you could ping a DC for example, but you cant RDP into it.  Or ping an internal webserver, but again cannot access its webpage.  Cant login to the phone system, or skype for business.  Many of it says subtype eq deny, and action is drop.... for applications like msrpc-base, ms-ds-smb-base, DNS, SSL, ntp, cotp.  However the source user IS correctly identified, and the allow rule for this user is high up, actually rule 7 in this case VPN Access - Information Technology and its allow domain admins, information technology, a vpn-it ad group, and a particular service account we sometimes use to rdp to a particular server.  The thing is the destination is any / any / any and application and service are both any / any.  So why would it deny on things like ssl.

Its random, like I will have Outlook open and lock my screen one day and the next morning unlock my screen - I'm still connected to the VPN, Outlook is up to date.  I'll work for an hour or two and then start getting connectivity issues.  GP says I'm connected... the internet works fine (we split tunnel) but I can't access anything internal. I can ping but nothing else really.  I can fire up another computer on Cisco Anyconnect just so I can see the firewall logs for my GP IP address, and my user name is still correct but its starting to deny applications that shouldn't be denied because we have it set to "any".  

We have a few testers and they seem to see that they log in and it goes from pre-logon to their user, GP says connected, but they can't access ANYTHING internal (except pinging things) for like 10 minutes!  We initially had the Gp tunnel rename timeout at 200 seconds so there's enough time to allow login scripts to run, but we tried changing it to 0 and still nothing.  We thought maybe a good working config would rename the tunnel WITHOUT disconnecting the user.

At first I was ok for some time and then I would get into this random "state" of ping only, and we did seem to track it down to cookie auth expiring and renewing, so we disabled the cookies and did not fix.

We then thought maybe its user id an its not thinking about moving from pre-logon to user or not detecting the user correct, so we tried that tweak above about disabling user id timeout.

Then I noticed I would have these strange connectivity issues right around a HIP check, so I disabled that.

Still nothing.  We can't get this thing stable and we are supposed to finally fully migrate on it from a 100% stable working Cisco ASA next weekend.  I can't move forward with that plan if we cant get 80 work from home users on a stable VPN.  We've been sitting on these PA-3220's for a year now and just cant get it going.  I'm almost regretting going with Palo and maybe I should have gone with our other contender, Fortigate.  Were at a loss here.

[ksauer507]

This may have partially been related to the timeout, but I don’t think that’s the whole issue.  Are users are randomly being detected as domain\username , which matches AD groups and users in policies, but then sometimes they are being detected as domain.com\username which does not match any policy so it’s denied.  Then I think it couldn’t send a HIP report (traffic denied) and caused that issue.  But amongst many other issues too.  We created a SAML transform in AzureAD and now the GP client shows domain\user instead of user@domain.com, and 95% of the user UD detections are correct on windows machines, but the first login user-ID gets domain.com\user from source vpn-client.  On a Mac it always detects as domain.com\user since they are not talking to DCs.  Weird because in the GP client the .com isn’t there in the settings UI in either GP 5.2.10 or the new GP 6.0.  For now we have a failsafe “all” users from vpn zone under our specific user based vpn zone.  This way macs can get on and windows users can auth to AD and DNS until they get a legitimate user.  Downside is until then, or if your using a machine not on our domain like a Mac, iPhone or vendor… we can’t control what they have access too.

I posted about it here:  https://live.paloaltonetworks.com/t5/globalprotect-discussions/not-using-appid-but-gp-connections-ar...

Would love to “normalize” everything from the Global Protect Enterprise Application in Azure to the GP portal as domain\user, as this would actually match AD users and groups and then fall into the correct policies.  Also across operating systems (windows, mac, iOS, etc).

[ksauer507]

This is prelimiary but for the first time ever we are seeing Mac and IOS clients now finally match the proper user-id and access policy.  What we changed was go into the certificate profile for the cert we are using for User-ID and change the User Domain Name from domain.com to domain.

In the help it does say netbios domain here so that obscure setting must have been what caused it.  Also on Windows machines, it seems after prelogon and login script runs, its correctly switches over to the domain user and attaches to the correct policy pretty consistently as of this writing.

PAN support was UNHELPFUL blaming Azure AD and take this issue over to the Azure AD team.  The claims in the SAML properties in Azure AD just have to match something you have in the group ID settings, and we chose userPrinicipalName, so in GP it looks like username@domain.colm.

[ksauer507]

No, not fixed.  After a few hours it went back to detecting users as domain.com\username, out of nowhere, and because of that extra .com in the username, it doesn’t match any policies.

For further discussion on our issue, which as stumped TAC support, please follow here:

https://live.paloaltonetworks.com/t5/globalprotect-discussions/not-using-appid-but-gp-connections-ar...