Slow o365 downloads

[drewdown]

Just deployed HA 3020s in APAC and users are complaining that downloading office 2016 is painful, slow and eventually times out. Having a hard time figuring out why though, logs in PA don't show anything dropping or getting denied and data filtering is set to alert.

This wasn't an issue prior when using ASAs and the only change was moving from ASAs to dual PAs.  Can someone help figure out what the problem is?

[kiwi]

Hi @drewdown,

Dual PA's as in Active-Active ? Maybe there's some assymetric routing in play.

Instead of checking the logs you might want to verify the global counters for a clue.

https://live.paloaltonetworks.com/t5/Learning-Articles/How-to-Troubleshoot-Using-Counters-via-the-CL...

Cheers !

-Kiwi

[drewdown]

@kiwi Active/passive

When attempting to download o2016 from the end users machine I only see a couple increments on that command:

flow_tcp_non_syn_drop                 620262        1 drop      flow      session   Packets dropped: non-SYN TCP without session match

That goes from 0 - 5 depending on how many times I run it.   And when I run a packet capture for all stages 'drop' gets no hits but still not sure what the problem is.  Takes 5 hours for this to eventually fail but it always does.  

[kiwi]

Hi @drewdown,

That counter is in most cases related to asymmetric routing and/or TCP reassembly issues.

As a temporary workaround you could allow the non-syn-tcp first packet and verify if it fixes the slowness for you ... I say temporary because it's a setting that's generally not recommended ! (Allowing non-SYN TCP traffic may prevent file blocking policies from working as expected in cases where the client and/or server connection is not set after the block occurs).

Further debugging would be required to find out the nature of this counter going up.

There are multiple discussions here on live that discuss non syn tcp.

An example : https://live.paloaltonetworks.com/t5/General-Topics/About-non-syn-tcp-option/m-p/36407#M26763

Additional info about this counter :

https://live.paloaltonetworks.com/t5/Configuration-Articles/SYN-ACK-Issues-with-Asymmetric-Routing/t...

https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Set-the-Palo-Alto-Networks-Firewa...

Cheers !

-Kiwi.

[drewdown]

@kiwi

Disabled it and having the end user try it again and I do see drops now.   Lots of retransmissions and connection resets whereas I didn't see any of that prior.  

Odd because before I disabled that I opened a case with support and provided a packet capture which showed no drops so of course they want to say its not the FW causing the problem.  Which is wrong because I can flip the route back to the ASA and it will work without issue.  This a trend with PA support with stuff like this and first response being 'not us.'  Saw a similar problem with oracle/rnow and in the end there I just routed that traffic out a different path bypassing the PAs as well.  Frustrating to say the least.  

[drewdown]

@kiwi Still fighting this even with PA support working it to no avail (going on 3 weeks now).  The only way I can get this to work through a HA Pair of 3020s is either via a specific rule with no app-ids, no url filtering, or using app override for all ports. 

One thing I have learned about PA in all of this is everything they do just makes it all that much harder to figure out why its not working right.  Hell to even see what URLs were being blocked PA had to change all the categories to alert instead of allow in the hopes of catching something that would explain the problem. 

[Brandon_Wertz]

---

@drewdown wrote:

One thing I have learned about PA in all of this is everything they do just makes it all that much harder to figure out why its not working right.  Hell to even see what URLs were being blocked PA had to change all the categories to alert instead of allow in the hopes of catching something that would explain the problem. 

---

This is where you as an admin need to do the due diligence as to the most appropriate settings for your environment.  Palo might suggest as a "best practice" to set settings to allow, but it's known it's not going to log the action.  If you ever want support staff to be able to identify what traffic is and what might not be going through a firewall you should always have the setting to alert.

Similarly, when Palo suggest only using "log on session end" in your security rules.  This will cause problems as say there's a long running FTP session multiple hours long.  You aren't going to have a log until the transaction done.  Meanwhile a user can be having a problem with the FTP transaction which is being caused by the firewall.  So logging on session start is another benefit. 

Well being notified of an issue then needing to go back and modify your firewall policy only to go back to the user and ask them to try the transaction again is a PITA.

It's our jobs as admins to understand what's most appropriate for our own environments.

[drewdown · Accepted solution]

Finally got this resolved, in the end it was this setting that was causing it to break:

 set deviceconfig setting ctd skip-block-http-range yes

Default its set to no but the only way you would know it was at fault is if you apply filters with src/dst IP and then check the counters for anything changing.  Once I set that command to yes O365 installed without issue.

Relevant article: https://live.paloaltonetworks.com/t5/Configuration-Articles/How-to-Block-Multi-thread-HTTP-Downloads...

[SShnap]

I have the same issue with Microsoft store on Win10, when the user tries to download application from the store.

On the logs some of the traffic shows as *.deploy.static.akamaitechnologies.com and the other as IP address of Microsoft.

the application is web-browsing and ms-store and it being allowed and flaged as decrypted but some of the sessions ends as aged-out.

I tried to use the command without success, when I remove that user from decyption is being able to install the application.

thank you.

SSnap.

[jhendersho]

Update to this article for 8.1 and above.
https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLjPCAW