Slow accessing file shares using Global Protect VPN client

[hcao]

Has anyone experienced an issue where accessing file shares from a Windows 2008 R2 is really slow, often showing the hour glass taking up several minutes or cancel and retry opening file shares multiple times again before it opens up, after establishng a VPN session using Global Protect VPN client v4.1.9-7 (same issue also happened with GP client v4.1.8)

Note that the user is able to ping internal DNS server - or any other internal servers - by NetBios, FQDN, or IP just fine.

I've already setup the Application Override for SMB traffic from the VPN zone to the server, and even check the box "Disable Server Inspection Response" from the VPN secuirty policy.  All seems to work fine for me, and I am able to access the file shares just fine without delay after GP is connected.  However, I've a number of users who simple keep getting the hours glass when click on the file shares from the shortcut on their computer desktop.

From those users who has experienced the slowness issue, I notice that their home network or hot spots is 192.168.1.xxx, while my home network is 172.25.x.x

I've tested disabling the "direct access to local network" in the GP gateway, and the user who's on home network of 192.168.1.xxx could access the file shares just fine, but that would kill their internet, since I've split-tunnel enabled already.

Re-route their home internet traffic through VPN tunnel is not a possible option at this time.

PAN: v8.0.12

GP: 4.1.9

Windows 7-64bit, Windows 10-64-bit

Greatly appreciated if anyone can help.

Thanks!

[BPry]

@hcao,

It kind of sounds like the client is attempting to find the requested resource on the local network prior to attempting to find it within your corporate network through the GP tunnel. If you can't enable a full tunnel you won't be left with many options to address this outside of one of you donig a re-scheme of your network to avoid the overlap. 

[hcao]

@BPry  Thanks for taking the time trying to help me out.

I know the issue is with Palo Alto, since I also use Cisco AnyConnect client to connec to Cisco ASA that resides in the same datacenter and network, and all staff were happy to use the ASA for VPN without any delay at all.  Unfortunately, the my ASA has reached its end-of-life.

Re-IP my work networks is not a possible option.  It will involve a lot of work and some headaches along the way....

Have a non-IT user to change their home network IP scheme ....no, I really do not want to support their home network ...

I'll be looking for a replacement Cisco VPN solution

[SARowe_NZ]

Hi hcao,

Just wondering if you found a solution to this?

We seem to be having the same issue but am unable to diagnose the root cause.

Thanks,
Shannon

[BPry]

@SARowe_NZ ,

Can you specify if you are running into OPs issue where your end users have overlapping addressing, or whether you simply are experiancing slow SMB access across GlobalProtect? 

[SARowe_NZ]

Hi,

It is slow performance over GP VPN, but seems very specific to smb traffic. Other traffic is fine.

Overlapping subnets is not an issue in this case - the local network is different to the server subnet where the smb traffic is delivered from.

It is consistently slow over both wired and wireless networks (as the underlying internet), but, interestingly, 4G / cellular data does not have the same issue.

I don't know how to explain that, as the wired and wireless networks we have tested with have no apparent commonality, they are all completely separate.

Thanks,

Shannon

[hcao]

@Shannon,

No, I've not yet found the solution to this issue, even after I upgraded my fileshare server from 2008R2 to 2016.

My PAN OS is v8.0.12 and I'll upgrade it to the latest v8.1.8 (as of now) soon and see if it will fix the issue.

Did the SMB slowness issue just happen to you recently or SMB has been working fine before? PAN OS version?

I will post back to let you know.

[BPry]

@SARowe_NZ,

Do you have split-tunnel enabled or are you tunneling all traffic? 

If you are running split-tunnel a possible answer to the different pattern would be if it needs to check the local-network for the host or not. Wireless clients are assigned a public address and don't have a local network to check, while the wired and wireless clients would look at the local-network first depending on your GlobalProtect Configuration. 

[SARowe_NZ]

Hi,

No, it is a full tunnel. I did test with split tunnel, and creating host routes in the tunnel to the affected servers (this should ensure it checks that before the local network) but the result is the same.

Note: even if this were the issue, the local network on the wired/wireless networks is not the same as the server network so it should not check these first.

Thanks,

Shannon

[BPry]

@SARowe_NZ,

That's certaintly odd and something that I would raise a ticket with TAC about so they can look at your log files and your firewall configuration to see if they can determine why it might be different between the three subset of users. The most likely culpriate I can think of off hand, outside of a GP bug, would be the following:

1) Differences in matching security policies.

• Does all of the traffic match the same security rule?
• Is the server response insepction enabled on one policy but not the other?
• If multiple policies for the different users are bing hit, do they have the same threat profile?

2) Decryption Policies:

• Does the traffic match the same decryption policies?

3) Different agent settings getting applied based on source location, and therefore the client behaves differently?

Of course all possible scenarios I can think of that would logically cause a performance difference between users would depend on the users getting different agent configurations when they connect to GP or hitting differnet policies. If everyone is falling under the same policies and they aren't hitting different policies then I would really open a TAC case so they can review the logs and hopefully explain the difference in performance. 

[SARowe_NZ]

Hey,

Thanks for taking the time to think through this and provide such detailed responses - I certainly appreciate it.

1) Differences in matching security policies.

• Does all of the traffic match the same security rule?
• Is the server response insepction enabled on one policy but not the other?
• If multiple policies for the different users are bing hit, do they have the same threat profile?
• SR: All traffic matches the same security policy.I actually tried creating a security policy with no filtering and DSRI enabled (I even created a custom app with app-override to bypass L7 processing). Same result, so does not seem to be related to filtering or traffic processing through the policy engine.

2) Decryption Policies:

• Does the traffic match the same decryption policies?
• SR: There is no decryption turned on in this scenario.

3) Different agent settings getting applied based on source location, and therefore the client behaves differently?

SR: Single portal/gateway so all clients get the same agent settings

I've logged a TAC case and will see what they say. Suspect I could be in for the long haul with this one...

Thanks again.

[SARowe_NZ]

For others benefit it looks like this may be related to a network adapter driver "feature" called NS Offload.

We'd originally eliminated this as it occurred on both wired and wireless connections, but through trial and error I've found that disabling this feature seems to have an immediate impact and resolves the performance issues.

It seems this feature is on both the wireless and wired NIC. Tried latest/different drivers but issue persists.

There is not much documentation on it: https://www.intel.com/content/www/us/en/support/articles/000005585/network-and-i-o/wireless-networki...

My assumption is it is causing the issue when traffic is "offloaded" from the tunnel adapter to the real physical adapter.

The 4G adapter does not have this feature which is why it was a non-issue when connecting the VPN over 4G.

[hcao]

- I had upgraded my file share server from Windows 2008 R2 to Windows 2016, but accessing file shares is still slow over Global Protect VPN

- Finally, I upgraded the PAN to 8.1.8-h5 and it seemed to have resovled this issue.  Accessing the file shares over GP is now working much faster than before.