Slow Google searches on 9.0

[Sec101]

Recently we changed to 9.0 code.   We are running decryption on our firewalls.   I've seen some very slow google searches recently, and a few errors when searching all while  using chrome.  Eventually the page will load the search if I wait long enough.   It's almost like chrome is failing the search at first, and then succeeding?   Wondering if this has something to do with http2 decryption?

[reaper]

See if you can set chrome in developer mode (or install a.http plugin) to track how many queries are being launched when running a.search. since google is all about tracking etc you're probably launching a handful of sessions that all need to be decrypted and some may wait for others to complete so google can optimize your 'ad' spread

You could try either blocking some ad categories or disabling decryption for them

[Sec101]

This is interesting.   is there a plugin you suggest that is best for this?

[reaper]

I used to use 'live http headers' bit that's been infested with malware some time ago

The built-in dev tool is pretty good though: hit shift+ctrl+J or option+command+J and then hit the "network" tab. Then see what happens if you do a Google search (mine currently shows 60 requests)

This will also help you visualize where the delay is (are all queries slow or are there 1/few that are slowing everything down)

[Sec101]

Just random ones, but it's the majority.   Seems you can replicate by closing the browser and relaunching it and googling something.  Even went as far as allowing web advertisements category for google-base, to see if that would help.   Same thing.   Edge surfs just fine, even on google.   Just chrome.   Anything to do with this?

https://live.paloaltonetworks.com/t5/general-topics/ssl-decryption-err-http2-inadequate-transport-se... 

[Sec101]

it's interesting, as I see really long "Stalled" - some up to 50 -60 seconds.   All of these searches have the "h2" protocol involved.   If I strip alpn, all returns to normal and I can't replicate it.   All of the issues starts with a fresh window of chrome, and the first search you enter.   Below of an h2 transaction for a simple google search:

[Dpeters1]

Did you get anywhere with this? I've got the same situation at one location, 9.1.5, so probably not firmware related. I found if you disable TLS inspection for a test user/computer the issue goes away, so probably something to do with inspection/decoders. Going to do some more investigation tomorrow. I did some PCAPs and it's very strange, there's just a massive delay in the response, but nothing actually dropped or blocked by the firewall. 

[VincentPresogna]

Just a shot in the dark here. If I remember right, Chrome uses QUIC. Maybe try blocking it and see what happens. I block it on my PA so I can decrypt the traffic. https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/decryption/define-traffic-to-decrypt.html

[Dpeters1]

very true, and a good shout. I'm 99% certain it's blocked on this deployment but could be slipping through the net or getting borked in some way ! I'll check that too. thank you!

[Dpeters1]

Hello again, managed to cram an hour of tinkering into the end of the day. 

Not quic. it's blocked at the firewall with an app rule, i also disabled quic in the chrome browser to rule it out totally.

However, did make some progress, looking like the http2 decoder on the firewall. We had some similar issues with totally different sites ages ago when 9.0 first came out. (feels like a lifetime ago now)

Created an inspection profile with the strip ALPM option checked, this functionally disabled http2 and forces it down to plain old http. 

Have put in a decrypt rule with URL a custom URL category only containing www.google.com and this option enabled. 

seems to have fixed it. I wonder if the OP has the same issue, would be cool to hear!

Will see how it goes, also going to do some rummaging around and see if it's an issue at other locations and the users just haven't reported anything. Certainly dont see it where I am, but we're PANOS 10.0.2 pilot site!