GlobalProtect Troubleshooting Tips:
Split Tunnel Domain & Applications and Exclude Video Traffic Features
Background
GlobalProtect with on-premise firewall is utilized by employees to securely connect to their enterprise environment and access their corporate applications. GlobalProtect supports Split Tunnel Domain & Applications and Exclude Video Traffic features to exclude certain bandwidth clogging applications and domains to help enterprises with business continuity during high Work From Home (WFH) scenarios because of a COVID-19 pandemic or any other type of calamity.
NOTE: Split-tunnel traffic is not inspected by next-generation firewall and, therefore, does not have the threat-protection offered by Palo Alto Networks. Hence, customers are advised to carefully review before enabling this feature and then decide whether the split tunnel meets their environment needs.
Objective
The objective of this document is to provide enterprise administrators with troubleshooting tips and tricks related to Split Tunnel Domain & Applications and Exclude Video Traffic features. This will help administrators during implementation and operational maintenance of these features. For a configuration guide of this feature, refer to Optimized Split Tunneling for GlobalProtect and GlobalProtect: Implement Split Tunnel Domain and Applications.
Verification and Troubleshooting
The following verification and troubleshooting steps are written with consideration of the configuration specified in GlobalProtect: Implement Split Tunnel Domain, Applications, Exclude Video Traffic Configuration and applies to any such configurations.
Split Tunnel Domain & Application
To verify and troubleshoot the split tunnel domain and application traffic features, you can utilize the following steps:
- First step is to verify whether the configuration on the gateway for ‘Split Tunnel Domain’ or ‘Split Application’ has been pushed correctly on the GlobalProtect app or not. This can be verified by collecting GlobalProtect logs. For steps on collecting GlobalProtect logs refer to: How to Collect Logs From GlobalProtect Clients.
- Within GlobalProtect logs bundle, review PanGPS.log and verify that based on the configuration on the gateway GlobalProtect receives:
- ‘Split Tunnel’ configuration:
<exclude-split-tunneling-domain>
<member>*.ringcentral.com</member>
</exclude-split-tunneling-domain>
- ‘Split Application' configuration:
<exclude-split-tunneling-application>
<member>%AppData%\Local\RingCentral\SoftPhoneApp\Softphone.exe</member>
<member>%AppData%\Local\RingCentral\SoftPhoneApp\SoftPhoneMapiBridge.exe</member>
<member>/Applications/RingCentral for Mac.app/Contents/MacOS/Softphone</member>
</exclude-split-tunneling-application>
- Within the GlobalProtect logs bundle, also review gpsplit.log (the equivalent file on the macOS is PanNExt.log) and see the split tunnel and application rules applied. In the logs below, we can see that ‘.ringcentral.com’ application is bound to physical interface en0. Thus, traffic for the RingCentral application will be excluded from the VPN tunnel. Here, Rule 0 to 3 corresponds to the IP address of the domain and application we have configured on the gateway.
gpsplit [0x52bc2520] :860 Rule 0: 1TCP v4 50.239.202.198 0 > 2PHY (83115)
gpsplit [0x52bc2520] :860 Rule 1: 3APP %AppData%\Local\RingCentral\SoftPhoneApp\SoftPhoneMapiBridge.exe > 2PHY (0)
gpsplit [0x52bc2520] :860 Rule 2: 3APP %AppData%\Local\RingCentral\SoftPhoneApp\Softphone.exe > 2PHY (0)
gpsplit [0x52bc2520] :860 Rule 3: 3APP /Applications/RingCentral for Mac.app/Contents/MacOS/Softphone > 2PHY (0)
gpsplit [0x5fd50a40] :933 0x59bc4620 binding to interface en0, index 3
NOTE: If an FQDN resolves to multiple IP addresses, all the IP addresses will be added to the exclude rules.
- Change the debug level to “Dump”, to make sure that PanGPS.log will contain the details related to split-tunnel functionality (Settings -> Troubleshooting -> Logging Level). Make sure to mark the time of the test (when the issue has been reproduced), along with the domain being accessed
- You can also verify the connection table on the client machine and confirm that specific application connections are going via physical interface and not the tunnel interface. On macOS, use ‘netstat -arn’ or 'lsof -n -i | grep <application>' command, and on a Windows machine, this ‘netstat -anob’ command can be used.
- We can also utilize 'whois' lookup utility to find the public IP address associated with specific domains or ISPs.
whois lookup for IP address
- For application visibility on Windows platforms, Microsoft Network Monitor can also be utilized. More information can be found in this article: Information about Network Monitor 3.
- To track traffic for a specific domain, enable wireshark (or tcpdump) packet captures on the client machine on the physical and tunnel (utun) interface. This is considered the most reliable method to track the traffic for specific domains. Always take packet captures for both physical and tunnel interface when reporting split-tunnel issues to Palo Alto Networks support.
On macOS, use tcpdump: sudo tcpdump -i all -k INP -w gptest.pcapng
Wireshark can be used for capturing the same on Windows
NOTE: Make sure to mark the time of the test (when the issue has been reproduced), along with the domain being accessed
- To find an IP address for a specific domain, resolve the IP address of the specific domain using nslookup as shown below. Apply the resulting IP address as a filter in wireshark.
$ nslookup ringcentral.com
Non-authoritative answer:
Name: ringcentral.com
Address: 216.146.46.11
Name: ringcentral.com
Address: 216.146.46.10.
- Verify that split-tunnel configuration is working as per the order of operation below where application exclude takes precedence over application include followed by domain exclude take precedence over domain include, and then Network traffic is excluded or included based on the specific access route.
GlobalProtect split tunnel order
- Split-tunneling rules only apply to TCP/UDP traffic, so ICMP/ping is not subject to split-tunneling rules. Do not use ping to test whether split-tunnel rules are applied
- For detailed Windows Kernel side logs, which allows us to see the interaction between GlobalProtect filter driver and the kernel, use DebugView, which can be found here: debugview
Run dbgview.exe as Administrator
"Enable Verbose Kernel Output" and Start "Capture Kernel" (Ctrl + K)
NOTE: This can generate large amount of logs and may also impact endpoint performance. Please enable this only when requested by Palo Alto TAC or engineering teams.
- On macOS: please also check whether GlobalProtect system extension is active using
$ systemextensionsctl list
--- com.apple.system_extension.network_extension
enabled active teamID bundleID (version) name [state]
* * PXPZ95SK77 com.paloaltonetworks.GlobalProtect.client.extension (5.2.5-66/1) GlobalProtectExtension [activated enabled]
Run sudo launchctl list | grep palo command to confirm the presence of NetworkExtension.com.paloaltonetworks.GlobalProtect.client.extension
3rd Party Interoperability:
- Check if there is a 3rd party product which can prevent GlobalProtect from properly using filters/extensions to perform split-tunnel operations. Most of the times, conflicts are found with DLP (Data Loss Prevention), AV/AM (Anti-Virus/Anti-Malware) and other VPN types of software. In these cases we need to investigate whether the issue is on the GlobalProtect side or 3rd party vendor.
- Application based exclusion will only affect the traffic generated directly by the named application. If the application is using another (system) process (for example through IPC) to facilitate a connection (such as svchost.exe), GlobalProtect filter will not capture it. Excluding such (system) a process is not advisable as it may be utilized by another non-related application, which can cause unintended consequences. In these cases we can take one of the two approaches:
- Check if the traffic bypassing the rules, we aren’t capturing due to aforementioned reasons with the Application based exclusion, needs DNS resolution before transmission. If this is the case, we may be able to exclude the leaking traffic using domain-based exclusions
- In case we can’t use domain-based exclusion (no corresponding DNS transaction), we have to rely on route exclusion; This implies that the application is using well-known IP subnets as a destination (depending on the application, list may be found on the Internet)
- Such behavior has been noted for some applications such as MS Teams, Skype etc. Please refer to GlobalProtect: Optimizing Office 365 Traffic for additional information
- macOS: Some applications are having connection issues when split-tunnel rules are applied using the new Apple System Extensions framework. Starting with GlobalProtect 5.1.4 and macOS 10.15.4 GlobalProtect switched, as a best practice, from legacy KEXT (Kernel Extensions) to the new System Extension framework. Apple is deprecating KEXT starting with the macOS Big Sur release (ref. About system extensions and macOS and Deprecated Kernel Extensions and System Extension Alternatives ). Please confirm with the 3rd party vendors on their support for the new Apple framework.
Exclude Video Traffic
To verify and troubleshoot exclude video traffic from the tunnel (Windows and macOS only) feature, you can utilize following steps:
- Verify whether the configuration you have on your gateway for ‘Exclude video traffic from the tunnel (Windows and macOS only)’ has been pushed correctly on the GlobalProtect or not. This can be verified by collecting GlobalProtect logs. For steps on collecting GlobalProtect logs, refer to this knowledge article: How to Collect Logs From GlobalProtect Clients.
- Within the GlobalProtect logs bundle, you can review PanGPS.log and verify that ‘Exclude video traffic from the tunnel (Windows and macOS only)’ configuration is received from the gateway as shown below:
<exclude-video-redirect>yes</exclude-video-redirect>
- The firewall will send a redirect message to GlobalProtect once it understands that the specific video application needs to be excluded from the VPN tunnel. In our example, we are excluding YouTube traffic. It determines the application as video based on the initial http/https request from the client, and it also matches the destination domain in the request with the one configured. Review of PanGPS.log file within the GlobalProtect logs bundle will confirm the video redirect message received by GlobalProtect client from the gateway. Same can be seen in the logs below:
Split tunneling is enabled: 0 include app, 2 exclude app, 0 include domain, 3 exclude domain, video-redirect yes
Debug(1732): SP set exclude ip 74.125.166.167, port 443 for video redirect
Debug(1732): SP set exclude ip 173.194.167.166, port 443 for video redirect
Debug(1732): SP set exclude ip 173.194.167.166, port 443 for video redirect
- On the firewall, you can filter the session based on a specific application by using command ‘show session all filter application <application-name>’. The example below is filtering ‘youtube-base’ application:
Admin view of PA-3260 in show session all filter application command
- Review the specific session details based on the output from Step 4 by using command ‘show session id <session id>’. Look for 'tracker stage firewall: split tunnel' in the session detail output, which confirms that the traffic is being excluded from the VPN tunnel.
- Browser verification can also be performed for HTTP 302 redirect response received from the gateway for the URL or video application, which we have excluded. In Chrome, Firefox, or Internet Explorer, you can utilize the Web Developer/Developer tools and Network option within them for such verification. HTTP 302 URL redirect message is seen under the status or result column when the gateway sends a redirect message. The below snapshot provides an example for Firefox Web Developer tool where under status column 302 redirect received from the gateway is seen for video playback.
Example of the Firefox Web Developer tool showing status column 302 results
