97% speed decrease on SMB traffic (PANOS 8.1)

[as-mg]

We're currently having some issues with ms-ds-smb (both v2 and v3) traffic on our PA-3020's (active/passive pair), where we are seeing a 97% speed decrease measured against direct traffic.

In order to determine the source of the issue, I have tried to disable server response inspection and all the security profiles, but I'm still getting speeds around 3-4MB/s. If I create an application override rule for tcp/445 I'm suddenly seeing around 100MB/s.

I don't really expect 100MB/s with threat protection enabled, but 3-4MB/s makes it seem like we're hitting a bug, and the firewall is far from overworked in terms of sessions and dataplane CPU usage.

Have anyone else had issues with SMB on PANOS 8.1? This has been for the last couple of versions, and we're currrently running 8.1.4.

[as-mg]

I just set up our PA-200 lab unit to do a basic test between two Windows 7 workstations, and noticed the same on PANOS 7.1.

Have anyone managed to speed up SMB transfers on PANOS, or do we just have to deal with this?

I'm getting anywhere between 4 to 8MB/s on the firewalls, and close to 100MB/s when doing the application override.

Could anyone share some benchmarks for their own production environments? 

[OtakarKlier]

Hello,

I havent seen this however its probably a good candidate for a support case if you dont have one opened already. As for benchmarks, I think you will get replies that are all over the board as everyone here probably has a different setup.

Regards,

[Rajesh12]

Hello @as-mg

We do faced similar performance issue with SMB traffic which was improved immediately when we applied app-override. This is what TAC have to say for it. Currently we are running with App-override in place. Suggest you to submit a support case to verify the same.

"Performance issues during file transfers, improved by App-Override".

* What about inspection is causing this
==> This little snippet from Engineering may help clear up why there is slowness:

"There are differences in the way SMB content is inspected compared to other protocols such as http, ftp that can lead to decreased throughput values. SMB decoder is unable to implement suspend since file transfers are done in a block-based manner, requiring continuous CTD inspection to follow the protocol on each block. Suspending only for one file could allow evasion for all subsequent files in the same session.
For SMB, we scan every payload for content inspection and does not have any offload mechanism. Hence the reason, it is recommended to implement application override for SMB to get better throughput values"

* Have other customers reported the same or is this a known issue
==> Yes other customers has seen this behavior. This is a expected behavior due to inspection of large no packets at this time as explained above. In future it is possible there could be some enhancements in future releases. You are welcome to submit "enhancement request" with your local SE in this regards.

* What can we do to improve performance other than disabling inspection since that defeats the purpose of next gen features
==> Instead of an app-override you can also attempt disabling DSRI to see if it provides increased throughput without the application override:
https://live.paloaltonetworks.com/t5/Management-Articles/How-to-Improve-Performance-for-Protocols-li...

[Sec101]

What version of 8.1 are you running....we may be running into an issue with SMB and slowness in 8.1.4   .....something about this inspection that firewall is slow at.

[agolob]

Hi,

We also have issues on all the locations where file server is used. Is there the same problem with AFP application?

We are running 8.1.12. Any news if this is resolved in verison 9.x?

[RREALICA]

this is what experienced:

1. we did not encounter the slowness on 8.1.7, running for months
2. we upgraded to 8.1.13, running for a couple of weeks, did not noticed the issue, it could have been there but we just did not notice it.
3. we upgraded to 9.0.8 and the smb traffic is much slower. applied the app-id override and this seems to have resolved the performance issues. none of the other suggestions was able to address the slowness except for the app-id override. 
4. the strange part is that we have multiple firewalls that in this case the smb traffic traverses and we only have to apply the app override on one of them. so fw1 has the app override, fw2 is processing the smb traffic as normal and this seems to be fine in our environment.

i do have a case open with tech to find more info.

[as-mg]

I've created support cases with this issue in the past, for different versions of PANOS and different hardware, and each time I've gotten the same response that goes like "SMB creates a lot of packets, which will slow down traffic when we need to inspect it" .

As mentioned in my previous post, app override is not an option for us, and we've tried disabling DSRI for all the rules which are using SMB without any large benefits. We're seeing around 15-20MB/s on gigabit links onthe current TAC recommended 8.1 release.

If anyone has found any way to resolve this issue, please share.