PAN-OS 8.0 Decryption Issue with Firefox and Chrome

[Gertjan-HFG]

After uprading my lab to pan-os 8.0 The forward Decryption failed when using Firefox and Chrome.

IE 11 en Edge still works.

For example when i go to www.google.com,

Chrome displays: www.google.com uses an unsupported protocol.
                               ERR_SSL_VERSION_OR_CIPHER_MISMATCH

Firefox:  Advanced info: SSL_ERROR_NO_CYPHER_OVERLAP

In the PA logs:  Session end Reason = decrypt-error.

PA continues the tradition to break decryption on new major releases 😉

Does anyone else have same issue?

[DavidMicallef]

Hi

Any news on this? Have been seeing strange behaviour very similar to this with Android 7.1 / Google / PAN 8.0.1. 

Thanks 

[Gareth_Dale]

David, we had the same issue after upgrading from 7.1.8 to 8.0.1 and after 24 hours it had resolved itself. Our TAC gave us a command that may help:

debug dataplane reset ssl-decrypt certificate-cache

I've asked them the question to whether this issue resurfaces after a reboot.

We also have issue with most of the Google apps including Play Store, the Wiki and Instagram app and decryption. I have a feeling most will have to bypass decryption at the OS level...

[DavidMicallef]

Thanks for this. 

Yes - have been chasing why Android phone (Nexus running Android 7.1.1)  when initially connected notifies me of no internet. Think this is now a connectivity check via SSL, if I place a decryption exception I can get past this hurdle so guessing something is pinned in Android. 

Also Google play no good, activity feed on Google Now/Assistant no good. Monitoring -> Logs -> Traffic I see session end as "Policy-Deny" on the decrypted traffic - category identified is search engines. 
Do not decrypt - search engines & content delivery networks and everything seems a great deal better. 

Ill keep at it. This is all on PAN 8.0.1 with SSL decryption enabled. 

++ update very interesting:-  https://android-developers.googleblog.com/2016/07/changes-to-trusted-certificate.html   looks like Android 7.1.1 has tightened up and possibly mitm no longer possible.

+++ https://serializethoughts.com/2016/09/10/905/

[jarbu]

Hello,

We have that problem on 8.0.1 but we do not use VM series but PA3020. DId you find workaround?

[Gareth_Dale]

We also have a 3020, the only workaround we have at the moment is the command I posted earlier (untested) or wait 24 hours and it started working.

In terms of Google, et al and their apps not working with decryption, this is currently with our TAC (including the links above from David) and a remote session due this morning...

[PaulBrock]

Hi gd@westbuckland.devon.sch.uk

Can i ask how did you go with TAC in regards to " Google, et al and their apps not working with decryption, 

I know people are saying wait 2h hous and some of the problem appears to be resolved , however you are saying some problems remain.

To all people who upgraded to 8.0 or 8.0.1 , would you recomened too people to upgrade or are you regreting the fact due to this decryption issue.

Paul

[Gareth_Dale]

The app decryption issue is specific to Android devices running 7 (and potentially any future ones). I tested on an Android 5 device yesterday and all apps work without a problem. This is due to changes in the trust level of client certificates on 7, plus the Android 6/Instagram certificate pinning change. This was the case with PA 7.1.x so upgrading to 8.0.x made no difference to this issue.

This is separate to the wait xx hours after upgrade issue in Chrome/FireBadger. In that regard, the command I posted earlier apparently doesn't resolve the issue, but the TAC confirmed that it didn't reappear after a reboot. Our Chrome/FireMongoose issue just resolved itself after waiting until the next morning.

Gareth

[DavidMicallef]

Hi Team 

Any progress on this?

thanks 

[BPry]

@DavidMicallef on the Android piece? That is and is going to be broken until Google either relaxes on the trust level of the client certs or otherwise releases a way to actually get this to function correctly. The ball at this point is really in their court to either leave it how it is currently or actually allow people to mitm their own clients traffic. 

[DavidMicallef]

Both really 🙂 wait hours before ssl plays ball nicely & Android 7 issue. 

With the Android issue does anyone know of a whitelist/s that can be used to disable inspection on Google ip ranges / URLs? Although this isn't enough as the vast number of cdn's that apps use will still cause issue. Short of disabling ssl inspection all together I am at a loss in this regard. 

Thanks

David 

[Gareth_Dale]

In terms of waiting for the SSL to wake up, our ticket was closed as it's working fine after the initial 24 hours and, hopefully, won't return after a reboot. They did reassure me it wouldn't return, but I'll believe it when I see it. I did make the point of letting Palo know that this was a bug and that it should be on the 'Known Issues' part of their release notes.

In terms of Android 7... yeah... a bit of a dead end! The only option is to disable decryption on a test device and use the apps in question. Check the monitor for URLs and add them to a bypass rule. For Google Play, I've got the following:

android.clients.google.com

*.gvt1.com

*.googleusercontent.com

*.ggpht.com

*.googleapis.com

Quite a big hole in decryption with some of those wildcards, but it's the simplest way of getting the app to work at the expense of visibility.

For us, this has got to go through our IT Committee to decide whether we continue to break these apps and tell users to use a web browser or we allow the apps through encrypted and lose all URL info...

[DavidMicallef]

Hi

With the Android 7 bit, still struggling on the initial connectivity check once joining wifi, connects fine but phone states "no internet". 

Tried excluding from decryption

clients3.google.com

connectivitycheck.android.com

connectivitycheck.gstatic.com

www.google.com/gen_204

but no luck. Only passes internet check if "www.google.com"  is bypassed from inspection. 

URL logs show traffic to 

connectivitycheck.gstatic.com/generate_204

www.google.com/gen_204

clients3.google.com/generate_204

Gareth have also added your suggested domains (thank-you) :- 

android.clients.google.com

*.gvt1.com

*.googleusercontent.com

*.ggpht.com

*.googleapis.com

If the phone does not pass this connectivity check data will be pushed out via cellular if available.

Thanks

David

[Gareth_Dale]

For the initial wireless connection, all we did was add the URL that you've got in your post to a bypass SSL decryption URL category:

connectivitycheck.gstatic.com

That worked for our setup, but we also had to add it, and another, in a 'Unauthenticated Allowed' URL category:

clients3.google.com

connectivitycheck.gstatic.com

This feeds into a security rule above the staff/students that allow certain URLs out without authenticating.

[DavidMicallef]

Thanks Gareth, unfortunately all allowed through. I'll keep at it, have a few packet captures to look at. I am testing on Android 7.1.1 to be exact. 

Regards 

David