Why is SMB traffic slow ?

[kiwi]

Every once in a while, there's a returning question on why SMB traffic is so slow. In this blog, I'll highlight a couple of solutions. Proposed by both community members and TAC engineers, several community members have found these useful and they've helped solve issues in the past.

Searching the LIVE community will already provide you with a couple of links featuring some related discussions and KB articles. These might already provide you a solution or give you some guidance to troubleshoot if you're experiencing this issue:

• 97% Speed Decrease On SMB Traffic PAN-OS 8.1 (discussion)
• How to Improve Performance for Protocols like SMB and FTP Without Application Override (KB article)
• GlobalProtect SMB Traffic Slowness (discussion)
• GlobalProtect SSL VPN Slow SMB Transfers  (discussion)

Allow me to first explain why SMB is a bit of a special protocol and why it's behaving the way it is:

SMB content is inspected differently compared to other protocols, like HTTP or FTP for example. The SMB decoder is unable to implement suspend since file transfers are done in a block-based manner, requiring continuous CTD inspection to follow the protocol on each block. Suspending only for one file could allow evasion for all subsequent files in the same session. ForSMB, every payload is scanned for content inspection and there is no offload mechanism to increase speed. This generally leads to a decreased throughput.

Reading the above already hints to a possible solution/workaround. You can disable content inspection by adding an app-override for this specific traffic, this will allow the session through using fast-path. This approach should be used only if other fail safes are in place, and only between trusted hosts:

Policies > Application Override 

Tips & Tricks: How to Create an Application Override 

In case an App Override is not possible because L-7 inspection is required, an alternative workaround would be to disable server response inspection (DSRI).  SMB traffic is very chatty by nature.  By default, traffic in both directions from client-to-server (C2S) and from server-to-client (S2C) will be inspected.  With DSRI, the firewall will only inspect C2S traffic !

Policies > Security > Actions

 Typically, DSRI is used in environments where internal servers are trusted !!

Note that DSRI is not limited to SMB traffic and can be used on other scenarios as well:

DotW: Using DSRI with the Palo Alto Networks firewall

Improving Performance of HTTP with DSRI 

Make sure you're on a recommended PAN-OS release to ensure any issues encountered are not caused by software. So check out the PAN-OS Software Release Guidance to confirm you're running the recommended PAN-OS release or if you are in need of an upgrade to rule out eventual bugs from previous software versions.

Sometimes you might have to dig somewhat deeper or even have Palo Alto Networks Support look into the issue.  Here are some additional debugging steps or troubleshooting tips you can look into:

• If you have the option, try different protocols (SCP, FTP, HTTP) to make sure that it's an SMB decoder issue.
• If you're testing with SMBv3, try disabling multichannel on the Windows server and client. Because of the way that SMBv3 multi-channel works in splitting up files Palo Alto Networks recommends disabling SMB multi-channel through the Windows PowerShell. For more information on this task, please refer to: technet.microsoft.com/en-us/library/dn610980(v=ws.11).aspx
• "show counter global filter delta yes" is your friend! While the file transfer is ongoing, run this command a couple of times and check if you see suspicious counter building up.
• Check if the packet descriptors (on-chip) are spiking during the file transfer. You want make sure that SMB sessions aren't using a high percentage of pkt buffers during the file transfer. Use the CLI command show running resource-monitor to verify this.
• Things to collect for analysis: "show session id (session id)" of a few of those sessions + generate tech support file.

Can you think of additional debugging/troubleshooting steps or workarounds for this popular topic? Please share your experience and possible solutions in the comments section below!

Thanks for taking time to read this blog.
If you enjoyed this, please hit the Like (thumb up) button, don't forget to subscribe to the LIVEcommunity Blog area.

Kiwi out!

[bspilde]

Great Post @kiwi!

Adding to this, I have some remediation options, contingencies, or questions to consider when doing DSRI (Disable Server Response Inspection):

1. Is the traffic secured by other methods without needing to decode the server response?
   1. Require Cortex XDR or other endpoint product scanning the file being copied.
      1. Do this by creating a HIP check on the VPN device and require a match on XDR being enabled and updated in a HIP Profile. 
      2. Then create a firewall policy configured with an application match policy for ms-ds-smb, ms-smb-v2, and if supported on your OS, ms-smb-v3. Include in this policy a requirement for the HIP profile created in the previous step.
   2. Now, consider this... if you use DSRI, consider the potential impact this might havee on vulnerability protection, not just file blocking and malware protection.
2. Are you doing file blocking?
   1. You may not be able to use DSRI if your company mandates DLP or other file blocking policies. This may be obvious but just to reiterate, copying a file from the server to a workstation would not be inspected/scanned with DSRI. This is one reason @Kiwi mentioned it should only be used for trusted servers.
3. What is a trusted server? Is a NAS trusted?
   1. If the communication is with a NAS, there may be no endpoint protection on the NAS itself. Thus, leaving you to defend your environment either with the inspection of the firewall or the endpoint transferring the files.
   2. My definition of a trusted server is one that is on a supported OS, regularly patched, vulnerability scanned, status monitored, and running endpoint protection EDR or preferably an XDR solution such as Cortex XDR.
      1. Do you get that with your NAS?
      2. Are you allowing SMBv3 prior to PAN-OS 8.1? If so, you are essentially allowing the SMBv3 protocol to bypass the decoder on your firewall.

Note on SMB as mentioned by @kiwi : 

SMB Improvements with WildFire Support

Firewall SMB support now includes SMBv3 (3.0, 3.0.2, and 3.1.1) and has additional threat detection and file identification capabilities, performance, and reliability across all versions of SMB. These improvements provide an additional layer of security for networks, such as data center deployments, network segments, and internal networks by allowing files transmitted using SMB to be forwarded to WildFire for analysis. Because of the way that SMBv3 multi-channel works in splitting up files, customers should disable the use of multi-channel file transfer for maximum protection and inspection of files. As a result, Palo Alto Networks recommends disabling SMB multi-channel through the Windows PowerShell. For more information on this task, please refer to: technet.microsoft.com/en-us/library/dn610980(v=ws.11).aspx

[kiliasov]

Folks, what are the performance implications of having SMB/SMBv3 with Threat Prevention enabled?

[albertoMS]

Hi mates!

Almost two years after this entry was first posted, we have this issue in a PA-5250 running PAN-OS 10.1.5-h2, and I've checked this hasn't been addressed in later versions either.

In our case, DSRI nor SMBv3 multi-channel file transfer disabling solves nothing, whereas Application Override works, we aren't confortable with SMB traffic flowing without threat prevention even between trusted hosts.

Has anyone found any other solution? Is this going to be addressed in coming PAN-OS version or is Application Override the definite solution?

Thanks and regards!

[eronko]

Keanu reeves and AI / ML will protect you , no time and resources from PA to fix 3+ years legacy core firewall os performance problems.