topic Re: SSL Decryption bug in PAN-OS 9.1.14 in General Topics

SSL Decryption bug in PAN-OS 9.1.14

LAS — Mon, 23 May 2022 17:58:45 GMT

I recently upgraded from panos 9.1.13-h3 to 9.1.14 then SSL decryption stopped working, in the traffic monitor there wasn't any decryption error but when i excluded a PC the internet worked

and it seams other people are also having the same issue (Reddit ), but its not in the known issue list until now

so i had to revert to the previous version and its working now fine

Re: SSL Decryption bug in PAN-OS 9.1.14

LAYER_8 — Mon, 23 May 2022 18:55:44 GMT

I don't know if this is your exact issue, but it seems as if we are tracking something internally. PAN-194219, the software packet buffers are depleting erroneously during HTTP/2 inspection only post 9.1.14 upgrade, and subsequently not decrypting.

I am now following the issue and will post updates / workarounds.

Re: SSL Decryption bug in PAN-OS 9.1.14

LAS — Tue, 24 May 2022 10:06:53 GMT

Ok thank you, will not update until the issue is resolved

Re: SSL Decryption bug in PAN-OS 9.1.14

joergriether — Tue, 24 May 2022 13:54:14 GMT

Same here. Rolling back now....I wonder why there is not any statement by PA. Or is it and I didn't find it?

Re: SSL Decryption bug in PAN-OS 9.1.14

LAYER_8 — Tue, 24 May 2022 17:32:27 GMT

The issue is escalated to the senior support queue, they've successfully recreated it in a lab and are now identifying root cause.

Re: SSL Decryption bug in PAN-OS 9.1.14

N2Z2 — Tue, 31 May 2022 07:39:42 GMT

Hi,

could you confirm that the problem is present in both RSA and ECDSA algorithm?

There's a know issue (PAN-83215) for ECDSA but I got the error with RSA.

Thanks

Re: SSL Decryption bug in PAN-OS 9.1.14

LAS — Tue, 31 May 2022 07:50:07 GMT

The certificate that i use is also RSA

Re: SSL Decryption bug in PAN-OS 9.1.14

LAYER_8 — Wed, 01 Jun 2022 16:58:50 GMT

Good news! Root cause identified, yes, RSA will get stumped here, too:

NGFW> debug dataplane show ssl-decrypt session 321122

Session 321122(local 321122), 1.0.0.2[50393]-->2.0.0.2[443]
Proxy Flow
        Index: 721716, Type: proxy, Tag: 321122, Dir: cts
        Rule: CRPNY-Decrypt
        Profile: 18F-Outbound-Decrypt-Office
        4 Packets Pending for L7 Proc
        TCP state
                Server Established
                MSS 1460
                DELACK timer is not on, RXMT timer is not on
                Recv Next:  581584158, Window: 49152, Scale: 0
                Send Next:  590184458, Window:  5792, Scale: 0
                Send Max :  590184458, Send Unack:  590184458
                Slow Start Threshold: 1073725440
                Congestion Window: 17520, RTT: 0 ticks
                # of Out-of-Order Pkts: 0
                # of Retrans: 0, # of DupAcks: 0
                # of Unsent Pkts: 0, # of Unacked Pkts: 0
        SSL State
                Protocol Version: TLS1.2
                Cipher : TLS_RSA_WITH_AES_256_GCM_SHA384

Peer Flow
        Index: 721703, Type: proxy, Tag: 321122, Dir: stc
        Rule: CRPNY-Decrypt
        Profile: 18F-Outbound-Decrypt-Office
        Is Closed
        4 Packets Pending for L7 Proc
        TCP state
                Client Closed Wait
                Can't recv
                MSS 1460
                DELACK timer is not on, RXMT timer is not on
                Recv Next:  590184464, Window: 49152, Scale: 0
                Send Next:  581583672, Window:  5792, Scale: 0
                Send Max :  581583672, Send Unack:  581583672
                Slow Start Threshold: 1073725440
                Congestion Window: 17520, RTT: 0 ticks
                # of Out-of-Order Pkts: 0
                # of Retrans: 0, # of DupAcks: 0
                # of Unsent Pkts: 0, # of Unacked Pkts: 0
        SSL State
                Protocol Version: TLS1.2
                Cipher : TLS_RSA_WITH_AES_256_GCM_SHA384

It appears we are seeing upgrade requests from servers through a TLS session, TLS1.1 requests to 1.2 or in some cases 1.3, this is not allowed per RFC. The firewall reads the request as a header frame, and then tries to extract sequential data from the incoming packets of the session. Since the session isn't terminated, the bogus packet(s) aren't dropped, session depletion happens.

We are currently researching how to terminate the session when this type of request comes in out of order.

Re: SSL Decryption bug in PAN-OS 9.1.14

LCMember4169 — Fri, 03 Jun 2022 12:45:23 GMT

Yep, we just had to roll our HA environment back to 9.1.13-h3 because of this issue (RSA certs). Too bad it wasn't listed in the Known Issues.

Re: SSL Decryption bug in PAN-OS 9.1.14

N2Z2 — Mon, 06 Jun 2022 09:08:27 GMT

It looks like that a workaround has been published or am I wrong?

PAN-194395

The firewall drops all decrypted outbound (SSL Forward Proxy) HTTP/2 traffic after you upgrade to PAN-OS 9.1.14. Dropping this traffic prevents users from loading HTTP/2 web pages and accessing websites that use HTTP/2.

Workaround: On the SSL Forward Proxy tab in the Decryption profile attached to the Decryption Policy rule that controls the HTTP/2 traffic, select Strip ALPN. When you Strip ALPN, the firewall negotiates HTTP/1.1 instead of HTTP/2.

It seems working in my environment.

Re: SSL Decryption bug in PAN-OS 9.1.14

LAS — Mon, 06 Jun 2022 14:15:15 GMT

i will wait then for when its fixed, as i don't want to downgrade HTTP/2 connections to HTTP/1.1 as a workaround

Re: SSL Decryption bug in PAN-OS 9.1.14

LAYER_8 — Tue, 07 Jun 2022 00:02:06 GMT

Indeed, it is now documented in the known issues. If your security requirements allow for HTTP1.1 / strip ALPN workaround that will suffice. If not, we are still researching the TLS upgrade requests and out of order queue.

Re: SSL Decryption bug in PAN-OS 9.1.14

LAYER_8 — Tue, 14 Jun 2022 00:59:13 GMT

Root cause identified, the fix is coming in 9.1.15-h1. Your choices are upgrade (when available), strip ALPN (if allowed), downgrade/stay on 9.1.13-h3. Final answer 🙂

Re: SSL Decryption bug in PAN-OS 9.1.14

jpan123 — Tue, 14 Jun 2022 14:24:24 GMT

What is your TLS version? can you show the profile you are using this recommendation did not work. I will attached my profile here.

Re: SSL Decryption bug in PAN-OS 9.1.14

N2Z2 — Tue, 14 Jun 2022 14:29:36 GMT

Hi,

Min 1.0, max version "Max"

The configuration is the same as in your screenshot

Re: SSL Decryption bug in PAN-OS 9.1.14

jpan123 — Tue, 14 Jun 2022 14:34:30 GMT

Not working to me. 😞

Re: SSL Decryption bug in PAN-OS 9.1.14

jpan123 — Tue, 14 Jun 2022 16:17:24 GMT

Any recommendation that I can try?

Re: SSL Decryption bug in PAN-OS 9.1.14

LAYER_8 — Tue, 14 Jun 2022 16:25:29 GMT

Have you both enabled strip ALPN?

Re: SSL Decryption bug in PAN-OS 9.1.14

N2Z2 — Tue, 14 Jun 2022 16:29:21 GMT

In my case yes

Re: SSL Decryption bug in PAN-OS 9.1.14

jpan123 — Tue, 14 Jun 2022 16:36:57 GMT

What do you mean both enabled?