Slow Google searches on 9.0

Announcements

Changes to the LIVEcommunity experience are coming soon... Here's what you need to know.

Reply
Sec101
L3 Networker

Slow Google searches on 9.0

Recently we changed to 9.0 code.   We are running decryption on our firewalls.   I've seen some very slow google searches recently, and a few errors when searching all while  using chrome.  Eventually the page will load the search if I wait long enough.   It's almost like chrome is failing the search at first, and then succeeding?   Wondering if this has something to do with http2 decryption?

reaper
L7 Applicator

See if you can set chrome in developer mode (or install a.http plugin) to track how many queries are being launched when running a.search. since google is all about tracking etc you're probably launching a handful of sessions that all need to be decrypted and some may wait for others to complete so google can optimize your 'ad' spread

 

You could try either blocking some ad categories or disabling decryption for them

Tom Piens - PANgurus.com
Like my answer? check out my book! amazon.com/dp/1789956374
Sec101
L3 Networker

This is interesting.   is there a plugin you suggest that is best for this?

reaper
L7 Applicator

I used to use 'live http headers' bit that's been infested with malware some time ago

 

The built-in dev tool is pretty good though: hit shift+ctrl+J or option+command+J and then hit the "network" tab. Then see what happens if you do a Google search (mine currently shows 60 requests)

 

This will also help you visualize where the delay is (are all queries slow or are there 1/few that are slowing everything down)

Tom Piens - PANgurus.com
Like my answer? check out my book! amazon.com/dp/1789956374
Sec101
L3 Networker

Just random ones, but it's the majority.   Seems you can replicate by closing the browser and relaunching it and googling something.  Even went as far as allowing web advertisements category for google-base, to see if that would help.   Same thing.   Edge surfs just fine, even on google.   Just chrome.   Anything to do with this?

https://live.paloaltonetworks.com/t5/general-topics/ssl-decryption-err-http2-inadequate-transport-se... 

Sec101
L3 Networker

it's interesting, as I see really long "Stalled" - some up to 50 -60 seconds.   All of these searches have the "h2" protocol involved.   If I strip alpn, all returns to normal and I can't replicate it.   All of the issues starts with a fresh window of chrome, and the first search you enter.   Below of an h2 transaction for a simple google search:

Sec101_0-1605141273790.png

 

Dpeters1
L2 Linker

Did you get anywhere with this? I've got the same situation at one location, 9.1.5, so probably not firmware related. I found if you disable TLS inspection for a test user/computer the issue goes away, so probably something to do with inspection/decoders. Going to do some more investigation tomorrow. I did some PCAPs and it's very strange, there's just a massive delay in the response, but nothing actually dropped or blocked by the firewall. 

VincentPresogna
L3 Networker

Just a shot in the dark here. If I remember right, Chrome uses QUIC. Maybe try blocking it and see what happens. I block it on my PA so I can decrypt the traffic. https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/decryption/define-traffic-to-decrypt.html

 

____________________

Just another I.T. Guy

Dpeters1
L2 Linker

very true, and a good shout. I'm 99% certain it's blocked on this deployment but could be slipping through the net or getting borked in some way ! I'll check that too. thank you!

Dpeters1
L2 Linker

Hello again, managed to cram an hour of tinkering into the end of the day. 

Not quic. it's blocked at the firewall with an app rule, i also disabled quic in the chrome browser to rule it out totally.

 

However, did make some progress, looking like the http2 decoder on the firewall. We had some similar issues with totally different sites ages ago when 9.0 first came out. (feels like a lifetime ago now)

 

Created an inspection profile with the strip ALPM option checked, this functionally disabled http2 and forces it down to plain old http. 

Dpeters1_0-1607018480446.png

 

Have put in a decrypt rule with URL a custom URL category only containing www.google.com and this option enabled. 

 

seems to have fixed it. I wonder if the OP has the same issue, would be cool to hear!

 

Will see how it goes, also going to do some rummaging around and see if it's an issue at other locations and the users just haven't reported anything. Certainly dont see it where I am, but we're PANOS 10.0.2 pilot site!

 

 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!