DNS Resolution stops after ~10s after connecting with GlobalProtect

[aimsnss]

Hello...

Many of my end users are now reporting that after approximately 10 minutes of logging to VPN using the GlobalProtect client they lose DHS resolution to internal and external resources. For example, when this happens. Users cannot access or even ping a server, by either its FQDN or by IP number. In addition, users also report they cannot access external resources such as Google. However, the GP client will still show "Connected." If the user disconnects/reconnects the GP client DNS resolution is restored. Then this cycle starts over again. This started several weeks ago and to my knowledge we have not made any changes to either the GP client or the FW. A support case, so far, has not resulted in a solution.

[SutareMayur]

Hi @aimsnss ,

Is it a split VPN tunnel ? Also what do you see in the traffic logs?

Mayur

[aimsnss]

Yes we are allowing split tunneling. Nothing in the traffic logs found so far. Tech support suggests putting the firewall into debug mode, Wireshark a client and see if we can recreate the issue. 

[SutareMayur]

Hi @aimsnss ,

Which DNS server you have configured under global protect settings? Is it internal?

Mayur

[aimsnss]

Yes, confirmed. Internal DNS address is in play.

[SutareMayur]

May be packet captures during issue can help to understand the root cause.

Mayur

[Abdul_Razaq]

Hi @aimsnss ,

is the issue resolved ?. mostly your DNS traffic policy might not be configured for logging, that could be the reason you don't have a traffic log entry.

Also, make sure the GP client IP is configured to access your internal DNS(as well as the network reachbilty) as you are using internal resolver IP in the GP configuration.

[JAuld]

I am having this issue and I cannot for the life of me work out whats going on. AFAIK im the only user experiencing this. I have a support ticket open with Arrow support and even they cannot work it out. Short of blowing away my windows installation, im not sure what else I can try. No one seems to have any solution online either, all forum posts end up dead with no solution found.

I'm using split DNS.

Everything could be working fine (all internal and external access working with no issues) for any random time between 10 seconds up to 4+ hours, then suddenly DNS cannot resolve anything internal or external. 

When the issue occurs, I cannot ping any internal resources at all whether by IP or FQDN.

I cannot ping any external resources by DNS, but CAN ping external by IP only. 

If I have any RDP sessions open to servers over the tunnel, they stay up, however I cannot create new connections to other servers. 

There's nothing in particular I can do to replicate the issue on demand. It just happens. 

There are no conflicts between my LAN or the routing table supplied by the tunnel. 

There is nothing changing my DNS (settings are consistent before and during the issue occurring.

I am running Windows 11 using GlobalProtect 6.0.0-262 and have tested 5.x versions too. All same issue.

This does not happen on my macbook on the same LAN. 

I have updated WiFi drivers, ethernet drivers, chipset drivers, etc. Anything I can think of. The issue occurs on WiFi and Ethernet.

Windows patches are all up to date.

Im just about to try removing Cortex XDR AV to rule that out. But apart from that, im all out of ideas. Can anyone shed some light on this please?

[BPry]

@JAuld,

Short of really diving into your logs and running a PCAP to see where that DNS traffic is actually getting sent, I don't think anyone will really be able to derive anything. The PCAP would allow you to at least see if the DNS traffic is even being sent and how Windows is routing things, and the PanGPS logs may point to an issue with the actual virtual adapter if that's the problem.

If this is static to one endpoint (meaning you can't reproduce this issue on another Windows endpoint when you login), re-imaging the device would be my go to solution for something like this instead of spending a ton of time trying to troubleshoot why it isn't working. 

[JAuld]

Hey mate,

Yes im thinkign im going to have to re-image. Removing cortex did nothing also. Its a shame coz i like to learn what the issues really are but this one has got me. Doesnt happen often coz im too stubborn lol. 

Here's some logs in case anyone has any ideas.  A dump from the service. 

(P4476-T8820)Dump ( 91): 03/30/22 07:21:02:983 Received DNS request for dns.msftncsi.com with type 1
(P4476-T8820)Dump ( 531): 03/30/22 07:21:02:983 EnforceSplitDns, ret1=-1, ret2=-1, type1=0, type2=0 (3/4-in/exclude), bReplyNoSuchName=0
(P4476-T8820)Dump ( 532): 03/30/22 07:21:02:983 EnforceSplitDns, qname=dns.msftncsi.com, from tunnel=1, reply no such name = 0
(P4476-T8820)Dump ( 942): 03/30/22 07:21:02:983 HandleDnsCallback result=passthrough
(P4476-T8820)Dump ( 718): 03/30/22 07:21:02:983 ST,lasterror is 997
(P4476-T8820)Dump ( 720): 03/30/22 07:21:02:983 ST,lasterror is ERROR_IO_PENDING
(P4476-T8820)Dump ( 723): 03/30/22 07:21:02:983 ST,write success
(P4476-T8820)Dump ( 871): 03/30/22 07:21:02:983 HandleDnsCallBack enter...
(P4476-T8820)Dump ( 916): 03/30/22 07:21:02:983 HandleDnsCallBack isPv6=0, from virtual interace=0
(P4476-T8820)Dump ( 925): 03/30/22 07:21:02:983 HandleDnsCallback: dns request..
(P4476-T8820)Dump ( 932): 03/30/22 07:21:02:983 HandleDnsCallback: initReplyBuffer done.
(P4476-T8820)Dump ( 91): 03/30/22 07:21:02:983 Received DNS request for dns.msftncsi.com with type 1
(P4476-T8820)Dump ( 531): 03/30/22 07:21:02:983 EnforceSplitDns, ret1=-1, ret2=-1, type1=0, type2=0 (3/4-in/exclude), bReplyNoSuchName=1
(P4476-T8820)Dump ( 532): 03/30/22 07:21:02:983 EnforceSplitDns, qname=dns.msftncsi.com, from tunnel=0, reply no such name = 1
(P4476-T8820)Dump ( 590): 03/30/22 07:21:02:983 EnforceSplitDns: Handle DNS request dns.msftncsi.com to server 203.12.160.35
(P4476-T8820)Dump ( 942): 03/30/22 07:21:02:983 HandleDnsCallback result=split dns
(P4476-T8820)Dump ( 561): 03/30/22 07:21:02:983 ST,READER, return reject DNS now with AWS
(P4476-T8820)Dump (3760): 03/30/22 07:21:02:983 ST,offset=34
(P4476-T8820)Dump (3417): 03/30/22 07:21:02:983 ST,ANSize = 76
(P4476-T8820)Dump (3423): 03/30/22 07:21:02:983 ST,return length=138
(P4476-T8820)Dump ( 718): 03/30/22 07:21:02:983 ST,lasterror is 997
(P4476-T8820)Dump ( 720): 03/30/22 07:21:02:983 ST,lasterror is ERROR_IO_PENDING
(P4476-T8820)Dump ( 723): 03/30/22 07:21:02:983 ST,write success
(P4476-T8820)Dump ( 871): 03/30/22 07:21:02:998 HandleDnsCallBack enter...
(P4476-T8820)Dump ( 916): 03/30/22 07:21:02:998 HandleDnsCallBack isPv6=0, from virtual interace=0
(P4476-T8820)Dump ( 925): 03/30/22 07:21:02:998 HandleDnsCallback: dns request..
(P4476-T8820)Dump ( 932): 03/30/22 07:21:02:998 HandleDnsCallback: initReplyBuffer done.
(P4476-T8820)Dump ( 91): 03/30/22 07:21:02:998 Received DNS request for dns.msftncsi.com with type 1
(P4476-T8820)Dump ( 531): 03/30/22 07:21:02:998 EnforceSplitDns, ret1=-1, ret2=-1, type1=0, type2=0 (3/4-in/exclude), bReplyNoSuchName=1
(P4476-T8820)Dump ( 532): 03/30/22 07:21:02:998 EnforceSplitDns, qname=dns.msftncsi.com, from tunnel=0, reply no such name = 1
(P4476-T8820)Dump ( 590): 03/30/22 07:21:02:998 EnforceSplitDns: Handle DNS request dns.msftncsi.com to server 203.12.160.36
(P4476-T8820)Dump ( 942): 03/30/22 07:21:02:998 HandleDnsCallback result=split dns
(P4476-T8820)Dump ( 561): 03/30/22 07:21:02:998 ST,READER, return reject DNS now with AWS
(P4476-T8820)Dump (3760): 03/30/22 07:21:02:998 ST,offset=34
(P4476-T8820)Dump (3417): 03/30/22 07:21:02:998 ST,ANSize = 76
(P4476-T8820)Dump (3423): 03/30/22 07:21:02:998 ST,return length=138
(P4476-T8820)Dump ( 718): 03/30/22 07:21:02:998 ST,lasterror is 997
(P4476-T8820)Dump ( 720): 03/30/22 07:21:02:998 ST,lasterror is ERROR_IO_PENDING
(P4476-T8820)Dump ( 723): 03/30/22 07:21:02:998 ST,write success
(P4476-T8820)Dump ( 871): 03/30/22 07:21:03:029 HandleDnsCallBack enter...
(P4476-T8820)Dump ( 916): 03/30/22 07:21:03:029 HandleDnsCallBack isPv6=0, from virtual interace=1
(P4476-T8820)Dump ( 925): 03/30/22 07:21:03:029 HandleDnsCallback: dns request..
(P4476-T8820)Dump ( 932): 03/30/22 07:21:03:029 HandleDnsCallback: initReplyBuffer done.
(P4476-T8820)Dump ( 91): 03/30/22 07:21:03:029 Received DNS request for dns.msftncsi.com with type 1
(P4476-T8820)Dump ( 531): 03/30/22 07:21:03:029 EnforceSplitDns, ret1=-1, ret2=-1, type1=0, type2=0 (3/4-in/exclude), bReplyNoSuchName=0
(P4476-T8820)Dump ( 532): 03/30/22 07:21:03:029 EnforceSplitDns, qname=dns.msftncsi.com, from tunnel=1, reply no such name = 0
(P4476-T8820)Dump ( 942): 03/30/22 07:21:03:029 HandleDnsCallback result=passthrough
(P4476-T8820)Dump ( 718): 03/30/22 07:21:03:029 ST,lasterror is 997
(P4476-T8820)Dump ( 720): 03/30/22 07:21:03:029 ST,lasterror is ERROR_IO_PENDING
(P4476-T8820)Dump ( 723): 03/30/22 07:21:03:029 ST,write success

[JAuld]

So I reimaged my machine today. Issue still exists. 

[JAuld]

I eventually worked this out. I was using RDP to connect to some servers using a different user account than that which I use for VPN authentication. The firewall was seeing this different user account coming from the same IP and denying connection attempts after that point, including DNS as GP is configured to use an internal DNS server for resolution.

After adding the other user account as a VPN user, this acted as an effective workaround. However I'm still unsure why this would not happen on my macbook, only on windows. I'm not sure if this is a bug or a feature, but the workaround has the unsatisfactory outcome of having multiple user accounts with VPN access allowed when they shouldnt be.

Any insight into how to fix this properly would be appreciated. 

Thanks