Split DNS Stoped working

[TheRealAndrewBrown]

Hi All!

Last week I was able to roll out split DNS to our production firewalls. This was tested successfully on a firewall in pre-prod and then moved to prod firewalls with same result.

Suddenly this morning queries to explicitly excluded domains are no longer being split. I've verified the configuration is good. I went back to pre-prod firewall and the same situation! This was a previously working config and no changes have been made since. I've verified this by using Wireshark on my local machine while connected to Global Protect. I don't see any DNS queries exiting the local adapter for excluded domains.

Previous to today I was able to see all DNS queries to excluded domains come through Wireshark when monitoring the local NIC.

Has anyone else experienced this problem?

[TheRealAndrewBrown]

Update:

Palo Alto support has confirmed the same behaviour in their lab environment. DNS requests to specific domains that are configured to be explicitly excluded from the VPN tunnel are still being tunnelled across Global Protect to the firewall.

This has been escalated within PA support for further investigation.

So for those of you fighting the same problem, this may be a bug. I will post another update as soon as I know more.

[TheRealAndrewBrown · Accepted solution]

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

• First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
• There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
• If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
• If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
• Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

If anyone from PA reads this forum please publish a complete guide on how to completely configure this feature. It would have saved me a lot of frustration and phone calls to support.

[TheRealAndrewBrown · Accepted solution]

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

• First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
• There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
• If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
• If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
• Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.