Assign private IP address failed

[stephen.mellor]

We buy GlobalProtect VPN as a service from a third party, for 1000+ users, and though the service worked well for some months our users have been plagued by login issues since the Christmas/New Year break - something changed, but we don't know what. The issue is that login fails with "Assign Private IP address failed". It's difficult to estimate the impact but this seems to happen for up to 10% of users at random - if they continue to try logging in they usually meet with success fairly quickly, but some users have been unable to log in for up to 2 hours.

Many of our users work from home, therefore this is quite a problem. Of course I have opened a support case with our provider, but they seem to be struggling. We have provided many troubleshooting logs from the client side, for successful and failed logins, but this seems to not be helping as the failure message is somewhat generic and there's no detail as to what has actually gone wrong.

We have a couple of client versions, all recent, eg 6.3.3-676. We can find no common factors with users that are seeing this error - we suspect that's it's actually all users, at random.

A quick google suggests that overlapping subnets can be a cause - that's definitely not the case here as our subnet is in the 10.0.0.0/8 space, and our users are typically connecting from home, with home routers usually providing IPs in the 192.168.0.0/16 space. Also, the failure is not permanent and repeated attempts can result in success, which rules this out.

We have a /22 allocation, so theoretically a maximum of 1021 simultaneous users. We typically see 700 users a day online, never more than 850. As is common with a VPN connection the clients report no DHCP lease time, as such, but presumably there is a DHCP server handing out IPs and presumably it has some sort of caching of IPs. Could this be a potential cause? What would the recommended settings be?

Clock errors have been cited as a possible cause - we've verified that our clients and authentication systems are all within spec - typically within tens of milliseconds, so that's not a problem.

What I'm looking for is clues as to what else may be the cause of such failures, and how we might be able to help our service provider to diagnose the issue. Unfortunately because we buy this as a service from a third party we don't have access to the back-end configuration or logs, but if we can point them in the right direction that would be great.

Thanks!

[JayGolf]

Hi @stephen.mellor ,

What is the exact /22 that is configured as an IP pool within the GlobalProtect Gateway config? I would ask your SP to provide the list of client pools they have configured. 

[BPry]

@stephen.mellor,

I don't think you're going to get a ton of help on this one because you don't know how the backend is actually configured. Whether they have GlobalProtect using a DHCP server to handle address assignment or if it's using IP Pools under GlobalProtect directly isn't really known here. 

Regardless of how they're handling address assignment, if they were hitting any address exhaustion that should be readily clear on the DHCP server side of things and from GlobalProtect on their end as well. I'd also just point out that you should have multiple pools available in the event that there is overlap. 

[stephen.mellor]

I think you're right, but at this point I'm trying anything. I think it won't be long before we abandon the service and roll our own solution - I'd really rather not though!

[stephen.mellor]

Thanks, I will do that. Interestingly we actually pay for slightly more capacity than /22 provides, and I have wondered if there are two pools defined in one part of the config, and just the one in another, and that IPs are occasionally being allocated from the second pool, which the rest of the config then can't handle. If that makes sense.

[OtakarKlier]

Hello,

Out of curiosity, what do you pay per user per month for the service? If you dont want to share publicly, you can DM me.

Regards,

[abayoumi21]

Could you please confirm the PAN-OS version running on the gateway? Some versions earlier than 11.2 have known issues that may affect IP address assignment in GlobalProtect, particularly when DHCP is used as the IP assignment method.

You may also want to review the GlobalProtect logs on the affected client machines, as these logs usually provide more detailed information about the failure stage during the connection process.

Additionally, capturing traffic on the client side using Wireshark could help identify whether DHCP communication is occurring correctly and at which stage the process fails.

As a recommendation, if DHCP is being used for IP allocation, consider creating an additional DHCP scope or segmenting the address pools to reduce potential allocation contention. In many deployments it is also preferable to use an external DHCP server to handle address assignment, as this can provide better scalability and easier troubleshooting.

[stephen.mellor]

Thanks, it looks like there's some things for me to look into there - the server-side info I'll pass on to our provider, and I'll look into the client-side stuff myself - I have had a good look at the client logs and haven't spotted anything relevant but I'll go through them again just to be sure. I personally have had this failure only a couple of times over two months, so wireshark is tricky, but maybe I can spend a morning repeatedly connecting / disconnecting, see if I can catch it. I'm intrigued that there are different methods for IP provision - I'll see if I can find some documentation on that, and I'll try to find out what our provider are using.