Troubleshoot Split Tunnel Domain & Applications and Exclude Video Traffic

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.
Audit
Last Reviewed: 08-10-2023 06:24 AM
Audited By: kiwi
L4 Transporter
100% helpful (9/9)

GlobalProtect Troubleshooting Tips:
Split Tunnel Domain & Applications and Exclude Video Traffic Features

 

 

Background

GlobalProtect with on-premise firewall is utilized by employees to securely connect to their enterprise environment and access their corporate applications. GlobalProtect supports Split Tunnel Domain & Applications and Exclude Video Traffic features to exclude certain bandwidth clogging applications and domains to help enterprises with business continuity during high Work From Home (WFH) scenarios because of a COVID-19 pandemic or any other type of calamity.

 

NOTE: Split-tunnel traffic is not inspected by next-generation firewall and, therefore, does not have the threat-protection offered by Palo Alto Networks. Hence, customers are advised to carefully review before enabling this feature and then decide whether the split tunnel meets their environment needs.

 

Objective

The objective of this document is to provide enterprise administrators with troubleshooting tips and tricks related to Split Tunnel Domain & Applications and Exclude Video Traffic features. This will help administrators during implementation and operational maintenance of these features. For a configuration guide of this feature, refer to Optimized Split Tunneling for GlobalProtect and GlobalProtect: Implement Split Tunnel Domain and Applications.

 

 

Verification and Troubleshooting

 

The following verification and troubleshooting steps are written with consideration of the configuration specified in GlobalProtect: Implement Split Tunnel Domain, Applications, Exclude Video Traffic Configuration and applies to any such configurations.

 

Split Tunnel Domain & Application

To verify and troubleshoot the split tunnel domain and application traffic features, you can utilize the following steps:

  1. First step is to verify whether the configuration on the gateway for ‘Split Tunnel Domain’ or ‘Split Application’ has been pushed correctly on the GlobalProtect app or not. This can be verified by collecting GlobalProtect logs. For steps on collecting GlobalProtect logs refer to: How to Collect Logs From GlobalProtect Clients.
  2. Within GlobalProtect logs bundle, review PanGPS.log and verify that based on the configuration on the gateway GlobalProtect receives:
    1. ‘Split Tunnel’ configuration:
      <exclude-split-tunneling-domain>
             <member>*.ringcentral.com</member>
      </exclude-split-tunneling-domain>​

       

    2. ‘Split Application' configuration:
      <exclude-split-tunneling-application>
             <member>%AppData%\Local\RingCentral\SoftPhoneApp\Softphone.exe</member>
             <member>%AppData%\Local\RingCentral\SoftPhoneApp\SoftPhoneMapiBridge.exe</member>
             <member>/Applications/RingCentral for Mac.app/Contents/MacOS/Softphone</member>
      </exclude-split-tunneling-application>​

       

  3. Within the GlobalProtect logs bundle, also review gpsplit.log (the equivalent file on the macOS is PanNExt.log) and see the split tunnel and application rules applied. In the logs below, we can see that ‘.ringcentral.com’ application is bound to physical interface en0. Thus, traffic for the RingCentral application will be excluded from the VPN tunnel. Here, Rule 0 to 3 corresponds to the IP address of the domain and application we have configured on the gateway.
    gpsplit [0x52bc2520] :860 Rule   0: 1TCP v4 50.239.202.198 0 > 2PHY (83115)
    gpsplit [0x52bc2520] :860 Rule   1: 3APP %AppData%\Local\RingCentral\SoftPhoneApp\SoftPhoneMapiBridge.exe > 2PHY (0)
    gpsplit [0x52bc2520] :860 Rule   2: 3APP %AppData%\Local\RingCentral\SoftPhoneApp\Softphone.exe > 2PHY (0)
    gpsplit [0x52bc2520] :860 Rule   3: 3APP /Applications/RingCentral for Mac.app/Contents/MacOS/Softphone > 2PHY (0)
    gpsplit [0x5fd50a40] :933 0x59bc4620 binding to interface en0, index 3
    NOTE: If an FQDN resolves to multiple IP addresses, all the IP addresses will be added to the exclude rules.

  4. Change the debug level to “Dump”, to make sure that PanGPS.log will contain the details related to split-tunnel functionality (Settings -> Troubleshooting -> Logging Level). Make sure to mark the time of the test (when the issue has been reproduced), along with the domain being accessed
  5. You can also verify the connection table on the client machine and confirm that specific application connections are going via physical interface and not the tunnel interface. On macOS, use ‘netstat -arn’ or 'lsof -n -i | grep <application>' command, and on a Windows machine, this ‘netstat -anob’ command can be used.
  6. We can also utilize 'whois' lookup utility to find the public IP address associated with specific domains or ISPs. 
    whois lookup for IP addresswhois lookup for IP address

  7. For application visibility on Windows platforms, Microsoft Network Monitor can also be utilized. More information can be found in this article: Information about Network Monitor 3.
  8. To track traffic for a specific domain, enable wireshark (or tcpdump) packet captures on the client machine on the physical and tunnel (utun) interface. This is considered the most reliable method to track the traffic for specific domains. Always take packet captures for both physical and tunnel interface when reporting split-tunnel issues to Palo Alto Networks support.
    On macOS, use tcpdump: sudo tcpdump -i all -k INP -w gptest.pcapng
    Wireshark can be used for capturing the same on Windows
    NOTE: Make sure to mark the time of the test (when the issue has been reproduced), along with the domain being accessed

  9. To find an IP address for a specific domain, resolve the IP address of the specific domain using nslookup as shown below. Apply the resulting IP address as a filter in wireshark.
    $ nslookup ringcentral.com
    Non-authoritative answer:
    Name: ringcentral.com
    Address: 216.146.46.11
    Name: ringcentral.com
    Address: 216.146.46.10.

  10. Verify that split-tunnel configuration is working as per the order of operation below where application exclude takes precedence over application include followed by domain exclude take precedence over domain include, and then Network traffic is excluded or included based on the specific access route.
    GlobalProtect split tunnel orderGlobalProtect split tunnel order
  11. Split-tunneling rules only apply to TCP/UDP traffic, so ICMP/ping is not subject to split-tunneling rules. Do not use ping to test whether split-tunnel rules are applied
  12. For detailed Windows Kernel side logs, which allows us to see the interaction between GlobalProtect filter driver and the kernel, use DebugView, which can be found here: debugview 
    Run dbgview.exe as Administrator
    "Enable Verbose Kernel Output" and Start "Capture Kernel" (Ctrl + K)
    NOTE: This can generate large amount of logs and may also impact endpoint performance. Please enable this only when requested by Palo Alto TAC or engineering teams.
  13. On macOS: please also check whether GlobalProtect system extension is active using
    $ systemextensionsctl list
    --- com.apple.system_extension.network_extension
    enabled active teamID bundleID (version) name [state]
    * * PXPZ95SK77 com.paloaltonetworks.GlobalProtect.client.extension (5.2.5-66/1) GlobalProtectExtension [activated enabled]
    Run sudo launchctl list | grep palo command to confirm the presence of NetworkExtension.com.paloaltonetworks.GlobalProtect.client.extension

3rd Party Interoperability:

  • Check if there is a 3rd party product which can prevent GlobalProtect from properly using filters/extensions to perform split-tunnel operations. Most of the times, conflicts are found with DLP (Data Loss Prevention), AV/AM (Anti-Virus/Anti-Malware) and other VPN types of software. In these cases we need to investigate whether the issue is on the GlobalProtect side or 3rd party vendor.
  • Application based exclusion will only affect the traffic generated directly by the named application. If the application is using another (system) process (for example through IPC) to facilitate a connection (such as svchost.exe), GlobalProtect filter will not capture it. Excluding such (system) a process is not advisable as it may be utilized by another non-related application, which can cause unintended consequences. In these cases we can take one of the two approaches:
    • Check if the traffic bypassing the rules, we aren’t capturing due to aforementioned reasons with the Application based exclusion, needs DNS resolution before transmission. If this is the case, we may be able to exclude the leaking traffic using domain-based exclusions
    • In case we can’t use domain-based exclusion (no corresponding DNS transaction), we have to rely on route exclusion; This implies that the application is using well-known IP subnets as a destination (depending on the application, list may be found on the Internet)
    • Such behavior has been noted for some applications such as MS Teams, Skype etc. Please refer to GlobalProtect: Optimizing Office 365 Traffic for additional information
  • macOS: Some applications are having connection issues when split-tunnel rules are applied using the new Apple System Extensions framework. Starting with GlobalProtect 5.1.4 and macOS 10.15.4 GlobalProtect switched, as a best practice, from legacy KEXT (Kernel Extensions) to the new System Extension framework. Apple is deprecating KEXT starting with the macOS Big Sur release (ref. About system extensions and macOS  and Deprecated Kernel Extensions and System Extension Alternatives ). Please confirm with the 3rd party vendors on their support for the new Apple framework.


Exclude Video Traffic

To verify and troubleshoot exclude video traffic from the tunnel (Windows and macOS only) feature, you can utilize following steps:

 

  1. Verify whether the configuration you have on your gateway for ‘Exclude video traffic from the tunnel (Windows and macOS only)’ has been pushed correctly on the GlobalProtect or not. This can be verified by collecting GlobalProtect logs. For steps on collecting GlobalProtect logs, refer to this knowledge article: How to Collect Logs From GlobalProtect Clients.
  2. Within the GlobalProtect logs bundle, you can review PanGPS.log and verify that ‘Exclude video traffic from the tunnel (Windows and macOS only)’ configuration is received from the gateway as shown below:
    <exclude-video-redirect>yes</exclude-video-redirect>​

     

  3. The firewall will send a redirect message to GlobalProtect once it understands that the specific video application needs to be excluded from the VPN tunnel. In our example, we are excluding YouTube traffic. It determines the application as video based on the initial http/https request from the client, and it also matches the destination domain in the request with the one configured. Review of PanGPS.log file within the GlobalProtect logs bundle will confirm the video redirect message received by GlobalProtect client from the gateway. Same can be seen in the logs below:
    Split tunneling is enabled: 0 include app, 2 exclude app, 0 include domain, 3 exclude domain, video-redirect yes
    Debug(1732): SP set exclude ip 74.125.166.167, port 443 for video redirect
    Debug(1732): SP set exclude ip 173.194.167.166, port 443 for video redirect
    Debug(1732): SP set exclude ip 173.194.167.166, port 443 for video redirect

  4. On the firewall, you can filter the session based on a specific application by using command ‘show session all filter application <application-name>’. The example below is filtering ‘youtube-base’ application:
    Admin view of PA-3260 in show session all filter application commandAdmin view of PA-3260 in show session all filter application command

  5. Review the specific session details based on the output from Step 4 by using command ‘show session id <session id>’. Look for 'tracker stage firewall: split tunnel' in the session detail output, which confirms that the traffic is being excluded from the VPN tunnel.
  6. Browser verification can also be performed for HTTP 302 redirect response received from the gateway for the URL or video application, which we have excluded. In Chrome, Firefox, or Internet Explorer, you can utilize the Web Developer/Developer tools and Network option within them for such verification. HTTP 302 URL redirect message is seen under the status or result column when the gateway sends a redirect message. The below snapshot provides an example for Firefox Web Developer tool where under status column 302 redirect received from the gateway is seen for video playback. 
    Example of the Firefox Web Developer tool showing status column 302 resultsExample of the Firefox Web Developer tool showing status column 302 results
Rate this article:
(2)
Comments
L0 Member

Recently ringcentral has changed where their directories reside. What was under %appdata% is now under %program files%. I noticed this after a number of users started complaining that ringcentral meetings would not work. I tried adding the following lines below to exclude from tunnel but still facing same issue. This article configuration worked well for the past year until recently after the folder relocation. If I remove exclusions from split tunnel it works great. Any ideas? Do I need a globalprotect license for this to work properly?

 

%programfiles%\RingCentral\RingCentral.exe
%programfiles(x86)%\RingCentralMeetings\bin\RingCentral_launcher.exe
%programfiles(x86)%\RingCentralMeetings\bin\RingCentralMeetings.exe

L2 Linker

In the GlobalProtect Log bundle, we don't have gpsplit.log -- just wondering where specifically is this file

"Within the GlobalProtect logs bundle, also review gpsplit.log (the equivalent file on the macOS is PanNExt.log) "

L4 Transporter

Hi @rajjair,

 

Comments in the following page can be helpful for you.

https://live.paloaltonetworks.com/t5/general-topics/globalprotect-working-from-home-prisma-access-an...

 

######

Thanks for the feedback. Actually gpsplit.log file is available as part of GlobalProtect logs bundle before GlobalProtect client 5.1.4 for macOS. After GlobalProtect client 5.1.4 and later, based on your macOS version you will either see gpsplit.log or PanNext.log [macOS 10.15.4 + GP 5.1.4 onwards]. For windows you can review PanGPS.log file. I will also update the document which you referred with this most current information. 

 

Thanks,

Nehal

######

L0 Member

Hi @nnaik ,

You sure Domain-based split tunneling is supposed to work for UDP traffic?

Same for the process based.

I did split-tunnel a domain, can see from the logs that an exception is added

 

""(P12804-T13544)Dump ( 797): 03/02/22 17:31:50:960 SP added an exclude ip 40.76.167.50, port 0, ttl 10 for domain GLOBAL.G.NSSVC.NET, original ttl=10, infinite ttl=no""

 

However UDP traffic still goes through the tunnel interface.

 

If I generate some random TCP traffic to the same domain (telnet on random port) I can confirm this TCP traffic goes through the physical interface.

L1 Bithead

These added domains, include or exclude, *.ringcentral.com, how does the client machine identifies it as to exit locally or go through the tunnel ?

 

One issue found is, when added a single domain in a split tunnel, some other domain traffic also starts coming through the tunnel.

L2 Linker

Thank you for correcting this article in regards to the log files and providing the details very helpful.

 

However, we seem to be running into an issue with this feature I am just wondering if this has been tested with a recent version of clients and if you guys have a link to try this feature out to confirm this is working as expected to not. In our environment we can see in the pangps.log we can see the video redirect being logged and the session on the firewall also shows the tracker stage of the split-tunnel occurring. However, the video errors out on the browser and the user are not able to play the content, based on the article there is 302 redirect but I don't see that occurring for us on the web trace, this video link is embedded on the site so wondering if this feature works with embedded videos or not.. Any information would help.. I have opened a ticket with support also on this

 

rajjair_0-1647144038150.png

 

I also tried to copy the URL and what I noticed on the first try was it fails on the second try it works after I refresh the browser.. but there is no 302 redirect as mentioned in the article above. below is a complete web trace when I have copied the original URL of the video and tried to play it.

 

rajjair_1-1647144173573.png

Thanks for any info anybody can share...

Raj

 

  • 44307 Views
  • 6 comments
  • 11 Likes
Register or Sign-in
Contributors
Article Dashboard
Version history
Last Updated:
‎01-14-2021 11:23 AM
Updated by: