SSL Inspection issues with GlobalProtect users

[Claw4609]

We're having some strange SSL/TLS Inspection errors while on GlobalProtect. We are getting unsupported-parameter errors while a user is connected to GlobalProtect trying to get to any internet site, including things like google.com. Doing a packet capture on the firewall it shows the connection trying to happen on tls 1.0 which we do indeed not allow as part of the decryption profile. This only happens while on GlobalProtect, other users with the same security policies and decryption polices applied work as intended and are decrypted as intended. 

This issue appears to have just started today, we updated PAN-OS to 10.2.9-h1 last week. This issue consistently happens on Chrome and Edge, but the issue seems to be almost non existent on Firefox. Which none of these browsers have tls 1.0 enabled. Even on the same computer, it works fine on-prem but has issues on GlobalProtect. Being its the weird combination of GlobalProtect users while using Chromium browsers, not sure which side is incorrectly acting on tls 1.0. I only see unsupported parameter or decryption error as the session end reason in the traffic logs, there are no errors in the decryption logs. Have tested on GP 6.0.7, 6.2.2, and 6.2.3, we are running PAN-OS 10.2.9-h1.

Unchecking the unsupported mode checks block fixes fixes the issue and gets us running for now.

Has anyone experienced something similar or a direction to look? We've also got a TAC case open.

[Jagdeep1]

Faced the same issue after upgrading to 10.2.8-h3. Also observed the same behavior on 10.2.9-h1 as well. Running GP version: 6.1.4, the same issue was on  5.1 version as well.

[Claw4609]

Palo sent this for a Prisma Access alert, however this appears to be the cause of the issue for our on-prem environment as well as I am able to replicate the issue on demand by editing this flag in the browser. If we change these flags in Chrome and Edge it resolves the issue. We for now have unchecked the "Block sessions with unsupported" checkbox in the SSL Decryption profile for the time being which is allowing us to get by without changing the flags on the endpoints at this time. We're working with TAC on what Palos recommendations are. 

Recommendations for Addressing Site Access Challenges with Decryption on Google Chrome Browser 124 and Higher

New incident: Monitoring For customers encountering challenges while accessing specific sites with decryption enabled and upon receiving "decrypt-unsupport-param" logs, particularly when using Google Chrome browser version 124 and higher, we suggest trying the following steps:

Issue Identification: The observed difficulties may arise from Kyber Support integrated by Chrome for the TLS 1.3 version.

Chrome Flags Configuration: Please review the configuration settings in Chrome Flags. This can be done by navigating to "chrome://flags/#enable-tls13-kyber" and examining the setup.

Disabling the Option: We encourage you to consider disabling the Kyber Support option and then relaunching the browser to assess if it resolves the issue.


If you continue to experience any difficulties, please open a support case, sincerely appreciate your patience as we diligently work to resolve this matter.

[Jagdeep1]

Thank You @Claw4609  for sharing.

[UtkarshKumar]

Hello Team

Is this a known issue, is anybody aware if this has been identified as bug?

[paul-norris]

We just started with the exact same behavior as OP on 4/20.  GP clients are the only ones affected... 10.2.8-h3 or 10.2.9-h1 ... any version of GP.  Disabling the Kyber flags fixes the issue as well as the other suggestions in this thread.

[jjhernandez · Accepted solution]

7/31/2024 Update: Updated ETA for 10.2.11, 11.2.2

7/15/2024 Update: Current ETA for 10.2.11, addl bug info.

5/14/2024 Update: See below - Bug ID and PANOS fixed versions.

5/6/2024 Update: See below.

Some additional info that might be useful:

• Chromium commit to "Enable PostQuantumKyber by default on desktop" occured on 3/12/2024
  • https://issues.chromium.org/issues/40910498
• Chromium Embedded Framework (CEF) was updated to Chromium v124 on 3/22/2024.
  • https://bitbucket.org/chromiumembedded/cef/src/master/
• Google Chrome enabled "TLS 1.3 hybridized Kyber support" in v124 on 4/16/2004.
  • https://developer.chrome.com/release-notes/124#x25519kyber768_key_encapsulation_for_tls 

Impacts:

• Any Chromium-based browser (Google Chrome, ARC, Brave, Opera, MS Edge, etc.) gets Kyber enabled by default.
  • Workaround: As noted above, "Disabling the Kyber flags fixes the issue" for now.
• Any applications that use the 3/22/2024 or later versions of the Chromium Embedded Framework (CEF) may also have Kyber on by default.
  • The macOS Slack Desktop App may be one of these apps. 
    • Workarounds: TBD.
    • These may be more challenging because these apps that embed CEF don't typically have the flags exposed.

5/6/2024 Update:

• Chromium Embedded Framework (CEF), Slack, and Kyber:
  • I have traffic logs of traffic from the macOS Slack Desktop app showing the typical decrypt-unsupport-param" errors seen with TLS1.3 traffic with Kyber enabled.
• SSL Decryption Workaround:
  • Per PAN TAC, the workaround in the SSL Decryption is to disable the following unsupported mode checks. This will allow all TLS1.3 packets with Kyber enabled to bypass SSL Decryption.
    • The upside is that users are functional.
    • The downside is that a significant (and growing) percentage of traffic is now bypassing SSL Decryption.
  • Unblock unsupported mode checks
  • This has an unintended consequence; Any traffic with unsupported SSL/TLS versions is now allowed to bypass SSL Decryption. So, you may also need to also broaden the min/max SSL/TLS Protocol versions in order to catch as much encrypted traffic as possible:
  • 

5/14/2024 Update

• Updates from my TAC case:
  • PAN ID: PAN-253546
  • Fixed versions: 11.2.2, 10.2.11, 10.1.14, 11.1.5, 11.0.7, 10.2.4-h19, 12.1.0

7/15/2024 & 7/31/2024 Update

• "The issue is caused when the large client hello is split into multiple packets and these arrive as out of order on the firewall." - PAN TAC.
  • Observation: This happened with Chromium-based browsers, including Google Chrome, on a very regular basis.
• 10.2.11, one of the fixed versions, is scheduled to ship around the end of July 2024 mid-August 2024.
  • Keep in mind that its software - it ships when PAN says its ready. The date here is an estimate only.
• Shipped: 11.2.2 shipped on 7/31/2024.
• Not Kyber related, but related and useful: As of PANOS 11.1, PAN firewalls can detect, block, and log the use of PQC and hybrid PQC algorithms in TLSv1.3 session...

[BWilson13]

With the hybridized kyber TLS 1.3 support enabled by Google which affects SSL decrypt, is this specifically impacting PAN-OS 10.1, 10.2 and 11.0? I ask out of curiosity because quantum security is in code 11.1 which may or may not be applicable in this scenario.

We're running 10.1.11-h4 and GP 6.0.7. 

[CReyna]

seeing the same issue with GP users in our environment. No issues when they are on prem or on prem wifi.
Disabling the chrome flag looks to resolve the issue. Thanks for the suggestion! 
This was a nasty one, took a good while to track down the very odd intermittent symptoms. 

Since this is only happening to users on GP, is there anything that palo can/will address on this? 

We are running PANOS - 10.1.9-H8 - GP 6.1.4

[Claw4609 · Accepted solution]

I have received this from our TAC case: 

We have internal reported issue and the target fix versions are:
10.2.11, 10.1.14, 11.1.5, 11.0.7

[BhamyShenoy]

Did you by any chance upgrade to the recommended version and test?

[Claw4609]

We're currently running 10.2.9-h1and other have reported the issue on other versions. The targeted fix releases arent out yet, given they are two versions away my guess would be 4-6 months until release but Im asking TAC if they have an ETA.

[BhamyShenoy]

Ah, my bad I did not look for 10.2 versions. Yes, I reported the issue with the TAC as well. We are on 10.1.11.  I am awaiting their response regarding fix versions and ETA. Thanks much! 

[MilesMclaughlin]

Is there a bug-id referenced in the case?

[Claw4609]

Not that they said at least, they just said they have an internal bug-id created. Also stated there is no current ETA on the targeted fix releases.