Split DNS Stoped working

TheRealAndrewBrown · ‎10-16-2020

Hi All!

Last week I was able to roll out split DNS to our production firewalls. This was tested successfully on a firewall in pre-prod and then moved to prod firewalls with same result.

Suddenly this morning queries to explicitly excluded domains are no longer being split. I've verified the configuration is good. I went back to pre-prod firewall and the same situation! This was a previously working config and no changes have been made since. I've verified this by using Wireshark on my local machine while connected to Global Protect. I don't see any DNS queries exiting the local adapter for excluded domains.

Previous to today I was able to see all DNS queries to excluded domains come through Wireshark when monitoring the local NIC.

Has anyone else experienced this problem?

TheRealAndrewBrown · ‎01-21-2021

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document.
There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

If anyone from PA reads this forum please publish a complete guide on how to completely configure this feature. It would have saved me a lot of frustration and phone calls to support.

View solution in original post

TheRealAndrewBrown · ‎01-21-2021

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document.
There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

View solution in original post

MP18 · ‎10-16-2020

@TheRealAndrewBrown

So far never saw this behaviour.

Which GP version you are running.

first thing I will do is pull the logs from the agent and check the GPS logs.

Regards

MP

TheRealAndrewBrown · ‎10-16-2020

Thanks MP18. I reviewed them this morning. While I see all the domains that are supposed to be excepted from the tunnel, I also see a lot of errors stating;

(T10796)Debug(1844): 10/15/20 00:33:28:575 Retry DnsQuery.
(T10796)Debug(1862): 10/15/20 00:33:28:575 Already takes 90 seconds for all dns queries.
(T10796)Debug(1864): 10/15/20 00:33:28:575 Exceeds 1 minute. Do not retry DnsQuery

I'm also seeing a lot of;

(T26916)Debug( 914): 10/15/20 08:25:19:427 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:20:031 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:46:423 HandleDnsCallback: failed to parse dns req packet.

However I have not been able to see in the logs what DNS server is unreachable or why it could not parse DNS packet. Still combing through the logs and I've logged a ticket with PA Support

MP18 · ‎10-16-2020

@TheRealAndrewBrown

Can you do nslookup for the gateway FQDN from the GP user.

External Gateway DNS server should be reachable and it should resolve from the FQDN for portal once GP agent is connected.

Regards

MP

TheRealAndrewBrown · ‎10-16-2020

@MP18

When connected to GP Gateway and I perform an nslookup to the external FQDN of the gateway (vpn.example.com). It doesn't resolve. Example.com is one of the domains that we tunnel through the VPN for resolution. However we don't have an internal DNS record on our AD servers for vpn.example.com I am a little unclear as to what you are asking for so I hope this was the right answer.

Split DNS Stoped working

Split DNS Stoped working

Show your appreciation!