topic Re: SSL Inspection issues with GlobalProtect users in General Topics

SSL Inspection issues with GlobalProtect users

Claw4609 — Mon, 22 Apr 2024 21:36:03 GMT

We're having some strange SSL/TLS Inspection errors while on GlobalProtect. We are getting unsupported-parameter errors while a user is connected to GlobalProtect trying to get to any internet site, including things like google.com. Doing a packet capture on the firewall it shows the connection trying to happen on tls 1.0 which we do indeed not allow as part of the decryption profile. This only happens while on GlobalProtect, other users with the same security policies and decryption polices applied work as intended and are decrypted as intended.

This issue appears to have just started today, we updated PAN-OS to 10.2.9-h1 last week. This issue consistently happens on Chrome and Edge, but the issue seems to be almost non existent on Firefox. Which none of these browsers have tls 1.0 enabled. Even on the same computer, it works fine on-prem but has issues on GlobalProtect. Being its the weird combination of GlobalProtect users while using Chromium browsers, not sure which side is incorrectly acting on tls 1.0. I only see unsupported parameter or decryption error as the session end reason in the traffic logs, there are no errors in the decryption logs. Have tested on GP 6.0.7, 6.2.2, and 6.2.3, we are running PAN-OS 10.2.9-h1.

Unchecking the unsupported mode checks block fixes fixes the issue and gets us running for now.

Has anyone experienced something similar or a direction to look? We've also got a TAC case open.

Re: SSL Inspection issues with GlobalProtect users

Jagdeep1 — Thu, 25 Apr 2024 19:12:11 GMT

Faced the same issue after upgrading to 10.2.8-h3. Also observed the same behavior on 10.2.9-h1 as well. Running GP version: 6.1.4, the same issue was on 5.1 version as well.

Re: SSL Inspection issues with GlobalProtect users

Claw4609 — Thu, 25 Apr 2024 19:17:26 GMT

Palo sent this for a Prisma Access alert, however this appears to be the cause of the issue for our on-prem environment as well as I am able to replicate the issue on demand by editing this flag in the browser. If we change these flags in Chrome and Edge it resolves the issue. We for now have unchecked the "Block sessions with unsupported" checkbox in the SSL Decryption profile for the time being which is allowing us to get by without changing the flags on the endpoints at this time. We're working with TAC on what Palos recommendations are.

Recommendations for Addressing Site Access Challenges with Decryption on Google Chrome Browser 124 and Higher

New incident: Monitoring

For customers encountering challenges while accessing specific sites with decryption enabled and upon receiving "decrypt-unsupport-param" logs, particularly when using Google Chrome browser version 124 and higher, we suggest trying the following steps:

Issue Identification: The observed difficulties may arise from Kyber Support integrated by Chrome for the TLS 1.3 version.

Chrome Flags Configuration: Please review the configuration settings in Chrome Flags. This can be done by navigating to "chrome://flags/#enable-tls13-kyber" and examining the setup.

Disabling the Option: We encourage you to consider disabling the Kyber Support option and then relaunching the browser to assess if it resolves the issue.

If you continue to experience any difficulties, please open a support case, sincerely appreciate your patience as we diligently work to resolve this matter.

Re: SSL Inspection issues with GlobalProtect users

Jagdeep1 — Thu, 25 Apr 2024 19:28:48 GMT

Thank You @Claw4609 for sharing.

Re: SSL Inspection issues with GlobalProtect users

UtkarshKumar — Fri, 26 Apr 2024 17:11:53 GMT

Hello Team

Is this a known issue, is anybody aware if this has been identified as bug?

Re: SSL Inspection issues with GlobalProtect users

paul-norris — Fri, 26 Apr 2024 19:12:39 GMT

We just started with the exact same behavior as OP on 4/20. GP clients are the only ones affected... 10.2.8-h3 or 10.2.9-h1 ... any version of GP. Disabling the Kyber flags fixes the issue as well as the other suggestions in this thread.

Re: SSL Inspection issues with GlobalProtect users

jjhernandez — Wed, 31 Jul 2024 20:30:23 GMT

7/31/2024 Update: Updated ETA for 10.2.11, 11.2.2

7/15/2024 Update: Current ETA for 10.2.11, addl bug info.

5/14/2024 Update: See below - Bug ID and PANOS fixed versions.

5/6/2024 Update: See below.

Some additional info that might be useful:

Chromium commit to "Enable PostQuantumKyber by default on desktop" occured on 3/12/2024
- https://issues.chromium.org/issues/40910498
Chromium Embedded Framework (CEF) was updated to Chromium v124 on 3/22/2024.
- https://bitbucket.org/chromiumembedded/cef/src/master/
Google Chrome enabled "TLS 1.3 hybridized Kyber support" in v124 on 4/16/2004.
- https://developer.chrome.com/release-notes/124#x25519kyber768_key_encapsulation_for_tls

Impacts:

Any Chromium-based browser (Google Chrome, ARC, Brave, Opera, MS Edge, etc.) gets Kyber enabled by default.
- Workaround: As noted above, "Disabling the Kyber flags fixes the issue" for now.
Any applications that use the 3/22/2024 or later versions of the Chromium Embedded Framework (CEF) may also have Kyber on by default.
- The macOS Slack Desktop App may be one of these apps.
  - Workarounds: TBD.
  - These may be more challenging because these apps that embed CEF don't typically have the flags exposed.

5/6/2024 Update:

Chromium Embedded Framework (CEF), Slack, and Kyber:
- I have traffic logs of traffic from the macOS Slack Desktop app showing the typical decrypt-unsupport-param" errors seen with TLS1.3 traffic with Kyber enabled.
SSL Decryption Workaround:
- Per PAN TAC, the workaround in the SSL Decryption is to disable the following unsupported mode checks. This will allow all TLS1.3 packets with Kyber enabled to bypass SSL Decryption.
  - The upside is that users are functional.
  - The downside is that a significant (and growing) percentage of traffic is now bypassing SSL Decryption.
- Unblock unsupported mode checks
- This has an unintended consequence; Any traffic with unsupported SSL/TLS versions is now allowed to bypass SSL Decryption. So, you may also need to also broaden the min/max SSL/TLS Protocol versions in order to catch as much encrypted traffic as possible:

5/14/2024 Update

Updates from my TAC case:
- PAN ID: PAN-253546
- Fixed versions: 11.2.2, 10.2.11, 10.1.14, 11.1.5, 11.0.7, 10.2.4-h19, 12.1.0

7/15/2024 & 7/31/2024 Update

"The issue is caused when the large client hello is split into multiple packets and these arrive as out of order on the firewall." - PAN TAC.
- Observation: This happened with Chromium-based browsers, including Google Chrome, on a very regular basis.
10.2.11, one of the fixed versions, is scheduled to ship around the ~~end of July 2024~~ mid-August 2024.
- Keep in mind that its software - it ships when PAN says its ready. The date here is an estimate only.
Shipped: 11.2.2 shipped on 7/31/2024.
Not Kyber related, but related and useful: As of PANOS 11.1, PAN firewalls can detect, block, and log the use of PQC and hybrid PQC algorithms in TLSv1.3 sessions.

Re: SSL Inspection issues with GlobalProtect users

BWilson13 — Wed, 01 May 2024 11:26:35 GMT

With the hybridized kyber TLS 1.3 support enabled by Google which affects SSL decrypt, is this specifically impacting PAN-OS 10.1, 10.2 and 11.0? I ask out of curiosity because quantum security is in code 11.1 which may or may not be applicable in this scenario.

We're running 10.1.11-h4 and GP 6.0.7.

Re: SSL Inspection issues with GlobalProtect users

CReyna — Fri, 03 May 2024 20:12:38 GMT

seeing the same issue with GP users in our environment. No issues when they are on prem or on prem wifi.
Disabling the chrome flag looks to resolve the issue. Thanks for the suggestion!
This was a nasty one, took a good while to track down the very odd intermittent symptoms.

Since this is only happening to users on GP, is there anything that palo can/will address on this?

We are running PANOS - 10.1.9-H8 - GP 6.1.4

Re: SSL Inspection issues with GlobalProtect users

Claw4609 — Sat, 04 May 2024 02:11:50 GMT

I have received this from our TAC case:

We have internal reported issue and the target fix versions are:
10.2.11, 10.1.14, 11.1.5, 11.0.7

Re: SSL Inspection issues with GlobalProtect users

BhamyShenoy — Mon, 06 May 2024 13:57:30 GMT

Did you by any chance upgrade to the recommended version and test?

Re: SSL Inspection issues with GlobalProtect users

Claw4609 — Mon, 06 May 2024 14:00:08 GMT

We're currently running 10.2.9-h1and other have reported the issue on other versions. The targeted fix releases arent out yet, given they are two versions away my guess would be 4-6 months until release but Im asking TAC if they have an ETA.

Re: SSL Inspection issues with GlobalProtect users

BhamyShenoy — Mon, 06 May 2024 14:41:25 GMT

Ah, my bad I did not look for 10.2 versions. Yes, I reported the issue with the TAC as well. We are on 10.1.11. I am awaiting their response regarding fix versions and ETA. Thanks much!

Re: SSL Inspection issues with GlobalProtect users

MilesMclaughlin — Wed, 08 May 2024 02:45:20 GMT

Is there a bug-id referenced in the case?

Re: SSL Inspection issues with GlobalProtect users

Claw4609 — Wed, 08 May 2024 12:01:12 GMT

Not that they said at least, they just said they have an internal bug-id created. Also stated there is no current ETA on the targeted fix releases.

Re: SSL Inspection issues with GlobalProtect users

TimBaker — Fri, 31 May 2024 08:50:16 GMT

PAN ID: PAN-253546

Fixed versions: 11.2.2, 10.2.11, 10.1.14, 11.1.5, 11.0.7, 10.2.4-h19, 12.1.0

10.1.14 dropped yesterday, with no mention in the release notes of this bug. Is it fixed in that version, or do we need to wait another couple months?

Re: SSL Inspection issues with GlobalProtect users

ADK999 — Mon, 03 Jun 2024 09:31:07 GMT

i'm not seeing anything related in the 1.1.14 release notes?
someone got it tested yet?

Re: SSL Inspection issues with GlobalProtect users

RomainSalmon — Fri, 14 Jun 2024 09:42:57 GMT

Hello,

Thanks for your post. What about the 10.2 ? Because there is not 10.2.10 yet... I did some wireshark capture and the palo alto is downgrading the protocol in my case from TLS1.3 to TLS1.2. This is one of the expected behaviour BUT why it is downgrading in TLS1.2 and not in TLS1.3 without the PQC algo??? I opened a TAC for that.

https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/decryption/post-quantum-cryptography-detection-and-control

Have a great day,

Re: SSL Inspection issues with GlobalProtect users

alpatt — Tue, 23 Jul 2024 07:41:18 GMT

Hi,

I also have a case open in regards to the issue. TAC confirmed the mentioned versions

Fixed versions: 11.2.2, 10.2.11, 10.1.14, 11.1.5, 11.0.7, 10.2.4-h19, 12.1.0

But we still have the issue in 10.1.14-h2 and I am also wondering why it is not mentioned in the release notes.

Re: SSL Inspection issues with GlobalProtect users

RomainSalmon — Tue, 23 Jul 2024 08:10:50 GMT

Hi, i am wondering the same thing.

I have seen this in the 11.2 release note : "Post-quantum cryptography (PQC) is all about the next-gen cryptographic algorithms. These babies replace the old-school ones like Diffie Hellman, RSA, and elliptic curve, which are sitting ducks for those quantum computers. With PAN-OS 11.2 Quasar, we're extending the post-quantum safe VPN introduced in PAN-OS 11.1 Cosmos by introducing PQC algorithms to create quantum-safe hybrid keys."

This functionality is available in 11.1: https://docs.paloaltonetworks.com/pan-os/11-1/pan-os-admin/decryption/post-quantum-cryptography-detection-and-control est celle de la version 11.1.There is no reference about PQC in previous versions.

FYI : 11.1.2-h3 is the preferred release.