Slow Google searches on 9.0

[Sec101]

Recently we changed to 9.0 code.   We are running decryption on our firewalls.   I've seen some very slow google searches recently, and a few errors when searching all while  using chrome.  Eventually the page will load the search if I wait long enough.   It's almost like chrome is failing the search at first, and then succeeding?   Wondering if this has something to do with http2 decryption?

[reaper]

See if you can set chrome in developer mode (or install a.http plugin) to track how many queries are being launched when running a.search. since google is all about tracking etc you're probably launching a handful of sessions that all need to be decrypted and some may wait for others to complete so google can optimize your 'ad' spread

You could try either blocking some ad categories or disabling decryption for them

[Sec101]

This is interesting.   is there a plugin you suggest that is best for this?

[reaper]

I used to use 'live http headers' bit that's been infested with malware some time ago

The built-in dev tool is pretty good though: hit shift+ctrl+J or option+command+J and then hit the "network" tab. Then see what happens if you do a Google search (mine currently shows 60 requests)

This will also help you visualize where the delay is (are all queries slow or are there 1/few that are slowing everything down)

[Sec101]

Just random ones, but it's the majority.   Seems you can replicate by closing the browser and relaunching it and googling something.  Even went as far as allowing web advertisements category for google-base, to see if that would help.   Same thing.   Edge surfs just fine, even on google.   Just chrome.   Anything to do with this?

https://live.paloaltonetworks.com/t5/general-topics/ssl-decryption-err-http2-inadequate-transport-se... 

[Sec101]

it's interesting, as I see really long "Stalled" - some up to 50 -60 seconds.   All of these searches have the "h2" protocol involved.   If I strip alpn, all returns to normal and I can't replicate it.   All of the issues starts with a fresh window of chrome, and the first search you enter.   Below of an h2 transaction for a simple google search:

[Dpeters1]

Did you get anywhere with this? I've got the same situation at one location, 9.1.5, so probably not firmware related. I found if you disable TLS inspection for a test user/computer the issue goes away, so probably something to do with inspection/decoders. Going to do some more investigation tomorrow. I did some PCAPs and it's very strange, there's just a massive delay in the response, but nothing actually dropped or blocked by the firewall. 

[VincentPresogna]

Just a shot in the dark here. If I remember right, Chrome uses QUIC. Maybe try blocking it and see what happens. I block it on my PA so I can decrypt the traffic. https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/decryption/define-traffic-to-decrypt.html

[Dpeters1]

very true, and a good shout. I'm 99% certain it's blocked on this deployment but could be slipping through the net or getting borked in some way ! I'll check that too. thank you!

[Dpeters1]

Hello again, managed to cram an hour of tinkering into the end of the day. 

Not quic. it's blocked at the firewall with an app rule, i also disabled quic in the chrome browser to rule it out totally.

However, did make some progress, looking like the http2 decoder on the firewall. We had some similar issues with totally different sites ages ago when 9.0 first came out. (feels like a lifetime ago now)

Created an inspection profile with the strip ALPM option checked, this functionally disabled http2 and forces it down to plain old http. 

Have put in a decrypt rule with URL a custom URL category only containing www.google.com and this option enabled. 

seems to have fixed it. I wonder if the OP has the same issue, would be cool to hear!

Will see how it goes, also going to do some rummaging around and see if it's an issue at other locations and the users just haven't reported anything. Certainly dont see it where I am, but we're PANOS 10.0.2 pilot site!

[Sec101]

It does seem that "strip alpn" fixes the slow responses.   It fixes it almost completely.   Yes- we do block quic, so maybe that is part of the issue?   

I'm surprised more people haven't seen/had this issue.   It does seem like strip ALPN fixes that - but isn't that downgrading the connection?   So what is the point of going to 9.0 with http/2 if you can't use it?   So does 10.0 fix this?   Too early for us to go that route.

[Dpeters1]

I think if you allowed quic, chrome would default to that for comms to google services (unless explicitly disabled in the browser settings), so that would also mitigate the issue, but you'd lose content control over all that traffic as the NGFW cant look inside quic at all.  I've always blocked quic on all our deployments for as long as I can remember. 

Well, there's 2 or 3 of us with it on the thread with the issue now! 🙂 

Tried hard to recreate in 10.0.2, cant do it. 

I've checked a location with 9.0.9, doesnt seem to do it their either, this ones a different client so slightly different policy setup, they were actually allowing quic, blocked it, still all working fine.

so strange, seems movable... 

they're all on the same apps/threats which includes the decoder updates. 

anyway feel like raising a TAC case? 😉

[tigeli]

I started seeing this on today with 9.1.6. Not just on Google but almost every site using http/2.

[Dpeters1]

Ouch, bad times. Such a weird issue how it's so intermittent between deployments. 

Have you tried the strip ALPN option on your decrypt profile ?

[tigeli]

Yeah I have, and it fixes the issue.