topic Re: Slow Google searches on 9.0 in General Topics

Slow Google searches on 9.0

Sec101 — Tue, 10 Nov 2020 17:34:53 GMT

Recently we changed to 9.0 code. We are running decryption on our firewalls. I've seen some very slow google searches recently, and a few errors when searching all while using chrome. Eventually the page will load the search if I wait long enough. It's almost like chrome is failing the search at first, and then succeeding? Wondering if this has something to do with http2 decryption?

Re: Slow Google searches on 9.0

reaper — Tue, 10 Nov 2020 21:28:16 GMT

See if you can set chrome in developer mode (or install a.http plugin) to track how many queries are being launched when running a.search. since google is all about tracking etc you're probably launching a handful of sessions that all need to be decrypted and some may wait for others to complete so google can optimize your 'ad' spread

You could try either blocking some ad categories or disabling decryption for them

Re: Slow Google searches on 9.0

Sec101 — Tue, 10 Nov 2020 22:46:13 GMT

This is interesting. is there a plugin you suggest that is best for this?

Re: Slow Google searches on 9.0

reaper — Tue, 10 Nov 2020 23:38:28 GMT

I used to use 'live http headers' bit that's been infested with malware some time ago

The built-in dev tool is pretty good though: hit shift+ctrl+J or option+command+J and then hit the "network" tab. Then see what happens if you do a Google search (mine currently shows 60 requests)

This will also help you visualize where the delay is (are all queries slow or are there 1/few that are slowing everything down)

Re: Slow Google searches on 9.0

Sec101 — Wed, 11 Nov 2020 02:35:27 GMT

Just random ones, but it's the majority. Seems you can replicate by closing the browser and relaunching it and googling something. Even went as far as allowing web advertisements category for google-base, to see if that would help. Same thing. Edge surfs just fine, even on google. Just chrome. Anything to do with this?

https://live.paloaltonetworks.com/t5/general-topics/ssl-decryption-err-http2-inadequate-transport-security/td-p/306467

Re: Slow Google searches on 9.0

Sec101 — Thu, 12 Nov 2020 00:37:30 GMT

it's interesting, as I see really long "Stalled" - some up to 50 -60 seconds. All of these searches have the "h2" protocol involved. If I strip alpn, all returns to normal and I can't replicate it. All of the issues starts with a fresh window of chrome, and the first search you enter. Below of an h2 transaction for a simple google search:

Re: Slow Google searches on 9.0

Dpeters1 — Thu, 03 Dec 2020 12:43:58 GMT

Did you get anywhere with this? I've got the same situation at one location, 9.1.5, so probably not firmware related. I found if you disable TLS inspection for a test user/computer the issue goes away, so probably something to do with inspection/decoders. Going to do some more investigation tomorrow. I did some PCAPs and it's very strange, there's just a massive delay in the response, but nothing actually dropped or blocked by the firewall.

Re: Slow Google searches on 9.0

VincentPresogna — Thu, 03 Dec 2020 16:51:40 GMT

Just a shot in the dark here. If I remember right, Chrome uses QUIC. Maybe try blocking it and see what happens. I block it on my PA so I can decrypt the traffic. https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/decryption/define-traffic-to-decrypt.html

Re: Slow Google searches on 9.0

Dpeters1 — Thu, 03 Dec 2020 16:54:40 GMT

very true, and a good shout. I'm 99% certain it's blocked on this deployment but could be slipping through the net or getting borked in some way ! I'll check that too. thank you!

Re: Slow Google searches on 9.0

Dpeters1 — Thu, 03 Dec 2020 18:03:48 GMT

Hello again, managed to cram an hour of tinkering into the end of the day.

Not quic. it's blocked at the firewall with an app rule, i also disabled quic in the chrome browser to rule it out totally.

However, did make some progress, looking like the http2 decoder on the firewall. We had some similar issues with totally different sites ages ago when 9.0 first came out. (feels like a lifetime ago now)

Created an inspection profile with the strip ALPM option checked, this functionally disabled http2 and forces it down to plain old http.

Have put in a decrypt rule with URL a custom URL category only containing www.google.com and this option enabled.

seems to have fixed it. I wonder if the OP has the same issue, would be cool to hear!

Will see how it goes, also going to do some rummaging around and see if it's an issue at other locations and the users just haven't reported anything. Certainly dont see it where I am, but we're PANOS 10.0.2 pilot site!

Re: Slow Google searches on 9.0

Sec101 — Thu, 03 Dec 2020 21:53:17 GMT

It does seem that "strip alpn" fixes the slow responses. It fixes it almost completely. Yes- we do block quic, so maybe that is part of the issue?

I'm surprised more people haven't seen/had this issue. It does seem like strip ALPN fixes that - but isn't that downgrading the connection? So what is the point of going to 9.0 with http/2 if you can't use it? So does 10.0 fix this? Too early for us to go that route.

Re: Slow Google searches on 9.0

Dpeters1 — Fri, 04 Dec 2020 14:09:32 GMT

I think if you allowed quic, chrome would default to that for comms to google services (unless explicitly disabled in the browser settings), so that would also mitigate the issue, but you'd lose content control over all that traffic as the NGFW cant look inside quic at all. I've always blocked quic on all our deployments for as long as I can remember.

Well, there's 2 or 3 of us with it on the thread with the issue now! 🙂

Tried hard to recreate in 10.0.2, cant do it.

I've checked a location with 9.0.9, doesnt seem to do it their either, this ones a different client so slightly different policy setup, they were actually allowing quic, blocked it, still all working fine.

so strange, seems movable...

they're all on the same apps/threats which includes the decoder updates.

anyway feel like raising a TAC case? 😉

Re: Slow Google searches on 9.0

tigeli — Tue, 08 Dec 2020 10:42:56 GMT

I started seeing this on today with 9.1.6. Not just on Google but almost every site using http/2.

Re: Slow Google searches on 9.0

Dpeters1 — Tue, 08 Dec 2020 11:11:46 GMT

Ouch, bad times. Such a weird issue how it's so intermittent between deployments.

Have you tried the strip ALPN option on your decrypt profile ?

Re: Slow Google searches on 9.0

tigeli — Tue, 08 Dec 2020 13:08:06 GMT

Yeah I have, and it fixes the issue.

Re: Slow Google searches on 9.0

Sec101 — Tue, 08 Dec 2020 14:34:27 GMT

I wonder why it makes for such a slow search on http/2 connections? It only appears to be on chrome too. Other browsers work fine.

Re: Slow Google searches on 9.0

tigeli — Wed, 09 Dec 2020 07:43:13 GMT

@Sec101 at least I could reproduce the issue on Safari, Chrome and Firefox.

Re: Slow Google searches on 9.0

Sec101 — Wed, 09 Dec 2020 17:31:06 GMT

you could reproduce on other browsers?

Re: Slow Google searches on 9.0

jmurphy — Wed, 09 Dec 2020 20:34:35 GMT

While waiting on TAC to call me back, I found this forum and tried the "workaround"...worked for me! So I guess now I can tell TAC what's actually wrong.

Re: Slow Google searches on 9.0

Dpeters1 — Thu, 10 Dec 2020 12:18:45 GMT

No, I wasnt able to re-produce in other browsers, although I didn't really try very hard, 99% over browser use is chrome. I've ended up adding a few youtube URLs and some related google video URLs to our "strip ALPN" url category, youtube videos were being very slow to buffer.

all a bit ironic really, HTTP/2 is support to be faster than 1.1 🙂

PLEASE PLEASE PLEASE let us know the outcome of your TAC call, I anticipate that we'll see an updated HTTP2 decoder in a future content update.

Speaking with some colleagues, they've seen the exact issue but with unrelated websites quite some time ago, and an updates HTTP2 decoder resolved the issue.