Split DNS Stoped working

[TheRealAndrewBrown]

Hi All!

Last week I was able to roll out split DNS to our production firewalls. This was tested successfully on a firewall in pre-prod and then moved to prod firewalls with same result.

Suddenly this morning queries to explicitly excluded domains are no longer being split. I've verified the configuration is good. I went back to pre-prod firewall and the same situation! This was a previously working config and no changes have been made since. I've verified this by using Wireshark on my local machine while connected to Global Protect. I don't see any DNS queries exiting the local adapter for excluded domains.

Previous to today I was able to see all DNS queries to excluded domains come through Wireshark when monitoring the local NIC.

Has anyone else experienced this problem?

[TheRealAndrewBrown · Accepted solution]

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

• First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
• There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
• If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
• If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
• Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

If anyone from PA reads this forum please publish a complete guide on how to completely configure this feature. It would have saved me a lot of frustration and phone calls to support.

[TheRealAndrewBrown · Accepted solution]

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

• First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
• There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
• If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
• If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
• Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

[MP18]

@TheRealAndrewBrown 

So far never saw this behaviour.

Which GP version you are running.

first thing I will do is pull the logs from the agent and check the GPS logs.

Regards

[TheRealAndrewBrown]

Thanks MP18. I reviewed them this morning. While I see all the domains that are supposed to be excepted from the tunnel, I also see a lot of errors stating;

(T10796)Debug(1844): 10/15/20 00:33:28:575 Retry DnsQuery.
(T10796)Debug(1862): 10/15/20 00:33:28:575 Already takes 90 seconds for all dns queries.
(T10796)Debug(1864): 10/15/20 00:33:28:575 Exceeds 1 minute. Do not retry DnsQuery

I'm also seeing a lot of;

(T26916)Debug( 914): 10/15/20 08:25:19:427 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:20:031 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:46:423 HandleDnsCallback: failed to parse dns req packet.

However I have not been able to see in the logs what DNS server is unreachable or why it could not parse DNS packet. Still combing through the logs and I've logged a ticket with PA Support

[MP18]

@TheRealAndrewBrown 

Can you do nslookup for the gateway FQDN from the GP user.

External Gateway DNS server should be reachable and it should resolve from the FQDN for portal  once GP agent is connected.

Regards

[TheRealAndrewBrown]

@MP18 

When connected to GP Gateway and I perform an nslookup to the external FQDN of the gateway (vpn.example.com). It doesn't resolve. Example.com is one of the domains that we tunnel through the VPN for resolution. However we don't have an internal DNS record on our AD servers for vpn.example.com I am a little unclear as to what you are asking for so I hope this was the right answer.

[MP18]

@TheRealAndrewBrown 

If you are pushing  DNS server info via Gateway to Agent then it should resolve the names for the FQDN and also for domains which you

wanna access via tunnel.

In our setup we have infoblox  in our internal DNS zone and we have configured Gateway with Internal DNS server IP.

All our servers or domains have records in the Internal zone.

Also Portal FQDN has DNS host record in Internal Zone of Infoblox.

When we connect to Gateway we can resolve all the domains and also Gateway address as our DNS server in Internal zone has all the records.

Regards

[TheRealAndrewBrown]

@MP18

Used infoblox in the past it was a neat tool...

As for Global Protect that's how I have it configured as well. All internal domains are set to be resolved by our internal AD servers. The only item we don't have configured is an internal record for vpn.example.com. However I fail to see how this could suddenly break the split tunnel DNS resolution.

Side note: I'm very surprised that GP only has an opt-out mechanism for DNS queries and it maxes at 300. Just to exclude all DNS entires for Office365 traffic is currently 111 entries. Generally I prefer to exclude all domains except internal corporate domains to save on bandwidth and provide better geo-location for DNS

[TheRealAndrewBrown]

Update:

After investigation we deemed this to be a licensing issue. After speaking our PA rep, he forwarded some documentation stating that split tunnel DNS is a licensed feature. In order to use it, you need to purchase the "Global Protect Gateway" License.

However after installing the appropriate license we attempted to roll forward last night and are still experiencing the same issues as initially noted. I have confirmed this is working on a PA-220 Lab unit but not a failover pair of PA-3050.

I'll continue to post updates as we progress.

Has anyone else seen this same behaviour?

[MP18]

@TheRealAndrewBrown 

Thanks for the update.

So far never seen this behaviour.

Regards

[TheRealAndrewBrown]

Thanks MP,

Apparently Palo Alto support has yet to see this problem either. They have been reviewing the configuration and tech support files as well as global protect log files from both the production environment and the lab environment.

I also had PA Support create a separate GP profile from scratch in the prod environment and it still failed.

(Side note: it IS working in the lab environment. I had made a mistake in testing the lab environment last week.)