Split DNS Stoped working

[TheRealAndrewBrown]

Hi All!

Last week I was able to roll out split DNS to our production firewalls. This was tested successfully on a firewall in pre-prod and then moved to prod firewalls with same result.

Suddenly this morning queries to explicitly excluded domains are no longer being split. I've verified the configuration is good. I went back to pre-prod firewall and the same situation! This was a previously working config and no changes have been made since. I've verified this by using Wireshark on my local machine while connected to Global Protect. I don't see any DNS queries exiting the local adapter for excluded domains.

Previous to today I was able to see all DNS queries to excluded domains come through Wireshark when monitoring the local NIC.

Has anyone else experienced this problem?

[MP18]

@TheRealAndrewBrown 

So far never saw this behaviour.

Which GP version you are running.

first thing I will do is pull the logs from the agent and check the GPS logs.

Regards

[TheRealAndrewBrown]

Thanks MP18. I reviewed them this morning. While I see all the domains that are supposed to be excepted from the tunnel, I also see a lot of errors stating;

(T10796)Debug(1844): 10/15/20 00:33:28:575 Retry DnsQuery.
(T10796)Debug(1862): 10/15/20 00:33:28:575 Already takes 90 seconds for all dns queries.
(T10796)Debug(1864): 10/15/20 00:33:28:575 Exceeds 1 minute. Do not retry DnsQuery

I'm also seeing a lot of;

(T26916)Debug( 914): 10/15/20 08:25:19:427 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:20:031 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:46:423 HandleDnsCallback: failed to parse dns req packet.

However I have not been able to see in the logs what DNS server is unreachable or why it could not parse DNS packet. Still combing through the logs and I've logged a ticket with PA Support

[MP18]

@TheRealAndrewBrown 

Can you do nslookup for the gateway FQDN from the GP user.

External Gateway DNS server should be reachable and it should resolve from the FQDN for portal  once GP agent is connected.

Regards

[TheRealAndrewBrown]

@MP18 

When connected to GP Gateway and I perform an nslookup to the external FQDN of the gateway (vpn.example.com). It doesn't resolve. Example.com is one of the domains that we tunnel through the VPN for resolution. However we don't have an internal DNS record on our AD servers for vpn.example.com I am a little unclear as to what you are asking for so I hope this was the right answer.

[MP18]

@TheRealAndrewBrown 

If you are pushing  DNS server info via Gateway to Agent then it should resolve the names for the FQDN and also for domains which you

wanna access via tunnel.

In our setup we have infoblox  in our internal DNS zone and we have configured Gateway with Internal DNS server IP.

All our servers or domains have records in the Internal zone.

Also Portal FQDN has DNS host record in Internal Zone of Infoblox.

When we connect to Gateway we can resolve all the domains and also Gateway address as our DNS server in Internal zone has all the records.

Regards

[TheRealAndrewBrown]

@MP18

Used infoblox in the past it was a neat tool...

As for Global Protect that's how I have it configured as well. All internal domains are set to be resolved by our internal AD servers. The only item we don't have configured is an internal record for vpn.example.com. However I fail to see how this could suddenly break the split tunnel DNS resolution.

Side note: I'm very surprised that GP only has an opt-out mechanism for DNS queries and it maxes at 300. Just to exclude all DNS entires for Office365 traffic is currently 111 entries. Generally I prefer to exclude all domains except internal corporate domains to save on bandwidth and provide better geo-location for DNS

[TheRealAndrewBrown]

Update:

After investigation we deemed this to be a licensing issue. After speaking our PA rep, he forwarded some documentation stating that split tunnel DNS is a licensed feature. In order to use it, you need to purchase the "Global Protect Gateway" License.

However after installing the appropriate license we attempted to roll forward last night and are still experiencing the same issues as initially noted. I have confirmed this is working on a PA-220 Lab unit but not a failover pair of PA-3050.

I'll continue to post updates as we progress.

Has anyone else seen this same behaviour?

[MP18]

@TheRealAndrewBrown 

Thanks for the update.

So far never seen this behaviour.

Regards

[TheRealAndrewBrown]

Thanks MP,

Apparently Palo Alto support has yet to see this problem either. They have been reviewing the configuration and tech support files as well as global protect log files from both the production environment and the lab environment.

I also had PA Support create a separate GP profile from scratch in the prod environment and it still failed.

(Side note: it IS working in the lab environment. I had made a mistake in testing the lab environment last week.)