Split DNS Stoped working

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Split DNS Stoped working

Hi All!

Last week I was able to roll out split DNS to our production firewalls. This was tested successfully on a firewall in pre-prod and then moved to prod firewalls with same result.

Suddenly this morning queries to explicitly excluded domains are no longer being split. I've verified the configuration is good. I went back to pre-prod firewall and the same situation! This was a previously working config and no changes have been made since. I've verified this by using Wireshark on my local machine while connected to Global Protect. I don't see any DNS queries exiting the local adapter for excluded domains.

Previous to today I was able to see all DNS queries to excluded domains come through Wireshark when monitoring the local NIC.

Has anyone else experienced this problem?

2 accepted solutions

Accepted Solutions

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

  • First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
  • There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
  • If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
  • If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
  • Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

 

If anyone from PA reads this forum please publish a complete guide on how to completely configure this feature. It would have saved me a lot of frustration and phone calls to support.

View solution in original post

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

  • First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
  • There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
  • If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
  • If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
  • Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

View solution in original post

13 REPLIES 13

Cyber Elite
Cyber Elite

@TheRealAndrewBrown 

 

So far never saw this behaviour.

Which GP version you are running.

 

first thing I will do is pull the logs from the agent and check the GPS logs.

 

Regards

 

MP

Help the community: Like helpful comments and mark solutions.

Thanks MP18. I reviewed them this morning. While I see all the domains that are supposed to be excepted from the tunnel, I also see a lot of errors stating;

 

(T10796)Debug(1844): 10/15/20 00:33:28:575 Retry DnsQuery.
(T10796)Debug(1862): 10/15/20 00:33:28:575 Already takes 90 seconds for all dns queries.
(T10796)Debug(1864): 10/15/20 00:33:28:575 Exceeds 1 minute. Do not retry DnsQuery

 

I'm also seeing a lot of;

 

(T26916)Debug( 914): 10/15/20 08:25:19:427 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:20:031 HandleDnsCallback: failed to parse dns req packet.
(T26916)Debug( 914): 10/15/20 08:25:46:423 HandleDnsCallback: failed to parse dns req packet.

 

However I have not been able to see in the logs what DNS server is unreachable or why it could not parse DNS packet. Still combing through the logs and I've logged a ticket with PA Support

@TheRealAndrewBrown 

 

Can you do nslookup for the gateway FQDN from the GP user.

External Gateway DNS server should be reachable and it should resolve from the FQDN for portal  once GP agent is connected.

 

Regards

MP

Help the community: Like helpful comments and mark solutions.

@MP18 

 

When connected to GP Gateway and I perform an nslookup to the external FQDN of the gateway (vpn.example.com). It doesn't resolve. Example.com is one of the domains that we tunnel through the VPN for resolution. However we don't have an internal DNS record on our AD servers for vpn.example.com I am a little unclear as to what you are asking for so I hope this was the right answer.

 

 

@TheRealAndrewBrown 

 

If you are pushing  DNS server info via Gateway to Agent then it should resolve the names for the FQDN and also for domains which you

wanna access via tunnel.

 

In our setup we have infoblox  in our internal DNS zone and we have configured Gateway with Internal DNS server IP.

All our servers or domains have records in the Internal zone.

Also Portal FQDN has DNS host record in Internal Zone of Infoblox.

 

When we connect to Gateway we can resolve all the domains and also Gateway address as our DNS server in Internal zone has all the records.

 

Regards

 

 

 

 

MP

Help the community: Like helpful comments and mark solutions.

@MP18

 

Used infoblox in the past it was a neat tool...

 

As for Global Protect that's how I have it configured as well. All internal domains are set to be resolved by our internal AD servers. The only item we don't have configured is an internal record for vpn.example.com. However I fail to see how this could suddenly break the split tunnel DNS resolution.

 

Side note: I'm very surprised that GP only has an opt-out mechanism for DNS queries and it maxes at 300. Just to exclude all DNS entires for Office365 traffic is currently 111 entries. Generally I prefer to exclude all domains except internal corporate domains to save on bandwidth and provide better geo-location for DNS

Update:

 

After investigation we deemed this to be a licensing issue. After speaking our PA rep, he forwarded some documentation stating that split tunnel DNS is a licensed feature. In order to use it, you need to purchase the "Global Protect Gateway" License.

 

However after installing the appropriate license we attempted to roll forward last night and are still experiencing the same issues as initially noted. I have confirmed this is working on a PA-220 Lab unit but not a failover pair of PA-3050.

 

I'll continue to post updates as we progress.

 

Has anyone else seen this same behaviour?

@TheRealAndrewBrown 

Thanks for the update.

So far never seen this behaviour.

 

Regards

 

MP

Help the community: Like helpful comments and mark solutions.

Thanks MP,

 

Apparently Palo Alto support has yet to see this problem either. They have been reviewing the configuration and tech support files as well as global protect log files from both the production environment and the lab environment.

 

I also had PA Support create a separate GP profile from scratch in the prod environment and it still failed.

 

(Side note: it IS working in the lab environment. I had made a mistake in testing the lab environment last week.)

Update:

Palo Alto support has confirmed the same behaviour in their lab environment. DNS requests to specific domains that are configured to be explicitly excluded from the VPN tunnel are still being tunnelled across Global Protect to the firewall.

This has been escalated within PA support for further investigation.

 

So for those of you fighting the same problem, this may be a bug. I will post another update as soon as I know more.

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

  • First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
  • There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
  • If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
  • If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
  • Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

 

If anyone from PA reads this forum please publish a complete guide on how to completely configure this feature. It would have saved me a lot of frustration and phone calls to support.

The issue was finally resolved in December and Split DNS is working as it should. However there was more than fix that was involved. For anyone stuck in the same situation I was, hopefully the information below will help.

  • First problem was licensing. while licensing is not required for the basic features of Global Protect to work, split DNS requires the Global Protect Gateway license. Without it, all DNS entries are forward to whatever DNS servers you configure on the gateway. No error is given when you configure it without a license and I could find no documentation about this license via google, or internal PA support documents. I had to speak to our rep who then sent me a document. 
  • There is a bug in Global Protect 5.2.2. The bug was sending causes the Global Protect client to send DNS queries out all local adapters including the VPN tunnel adapter on the user's computer. Upgrading to 5.2.3 resolved this problem
  • If you want to exclude all traffic from the VPN tunnel with the exception of your internal IP ranges and internal DNS records, include those items in your "included" items for both the Access Route and "Include Domain". Next, leave the exclude columns empty for both Access route and "Include Domain". This will ensure all public Internet traffic and DNS lookups will go out the local NIC on the user's computer.
  • If you're not using IPv6, disable it on the end user's computer. Global Protect will prefer IPv6 for DNS lookups. Some of our user's had IPv6 enabled on their internal home network and Global Protect began sending DNS queries for internal corporate records over IPv6 on the local NIC instead of over the VPN tunnel to the corporate DNS servers
  • Ensure in the "App config" in the portal that "Resolve All FQDNs Using DNS Servers Assigned by the Tunnel (Windows Only)" option is set to "no".

Palo Alto has thus far done a poor job on the documentation to implement split DNS. About 1/3 of information is spread out across multiple documents which can be hard to track down. The remaining 2/3s of the information needed to configure this required a support ticket to Palo Alto in order to get he full picture.

Hello

 

Thank you for the detailed steps in getting Split Tunnel working, I am stuck at the same part but I am running OSX on Macbooks and this option does state (windows only) do you know how this is achieved with OSX. 

 

I've configured everything as you mentioned (except the App feature for all FQDNS,since it's Windows Only)

But when connected to VPN, I see that all public queries done with nslookup are done by the configured DNS Servers in the GP Settings.

 

Thank you and best regards

Duque

There's no home like 127.0.0.1
  • 2 accepted solutions
  • 15620 Views
  • 13 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!