SSL Decryption bug in PAN-OS 9.1.14

[LAS]

I recently upgraded from panos 9.1.13-h3 to 9.1.14 then SSL decryption stopped working, in the traffic monitor there wasn't any decryption error but when i excluded a PC the internet worked

and it seams other people are also having the same issue (Reddit ), but its not in the known issue list until now

so i had to revert to the previous version and its working now fine

[LAYER_8]

I don't know if this is your exact issue, but it seems as if we are tracking something internally. PAN-194219, the software packet buffers are depleting erroneously during HTTP/2 inspection only post 9.1.14 upgrade, and subsequently not decrypting.  

I am now following the issue and will post updates / workarounds. 

[LAS]

Ok thank you, will not update until the issue is resolved

[joergriether]

Same here. Rolling back now....I wonder why there is not any statement by PA. Or is it and I didn't find it? 

[LAYER_8]

The issue is escalated to the senior support queue, they've successfully recreated it in a lab and are now identifying root cause. 

[N2Z2]

Hi,

could you confirm that the problem is present in both RSA and ECDSA algorithm?

There's a know issue (PAN-83215) for ECDSA but I got the error with RSA.

Thanks

[LAS]

The certificate that i use is also RSA

[LAYER_8]

Good news! Root cause identified, yes, RSA will get stumped here, too:

NGFW> debug dataplane show ssl-decrypt session 321122

Session 321122(local 321122), 1.0.0.2[50393]-->2.0.0.2[443]
Proxy Flow
        Index: 721716, Type: proxy, Tag: 321122, Dir: cts
        Rule: CRPNY-Decrypt
        Profile: 18F-Outbound-Decrypt-Office
        4 Packets Pending for L7 Proc
        TCP state
                Server Established
                MSS 1460
                DELACK timer is not on, RXMT timer is not on
                Recv Next:  581584158, Window: 49152, Scale: 0
                Send Next:  590184458, Window:  5792, Scale: 0
                Send Max :  590184458, Send Unack:  590184458
                Slow Start Threshold: 1073725440
                Congestion Window: 17520, RTT: 0 ticks
                # of Out-of-Order Pkts: 0
                # of Retrans: 0, # of DupAcks: 0
                # of Unsent Pkts: 0, # of Unacked Pkts: 0
        SSL State
                Protocol Version: TLS1.2
                Cipher : TLS_RSA_WITH_AES_256_GCM_SHA384

Peer Flow
        Index: 721703, Type: proxy, Tag: 321122, Dir: stc
        Rule: CRPNY-Decrypt
        Profile: 18F-Outbound-Decrypt-Office
        Is Closed
        4 Packets Pending for L7 Proc
        TCP state
                Client Closed Wait
                Can't recv
                MSS 1460
                DELACK timer is not on, RXMT timer is not on
                Recv Next:  590184464, Window: 49152, Scale: 0
                Send Next:  581583672, Window:  5792, Scale: 0
                Send Max :  581583672, Send Unack:  581583672
                Slow Start Threshold: 1073725440
                Congestion Window: 17520, RTT: 0 ticks
                # of Out-of-Order Pkts: 0
                # of Retrans: 0, # of DupAcks: 0
                # of Unsent Pkts: 0, # of Unacked Pkts: 0
        SSL State
                Protocol Version: TLS1.2
                Cipher : TLS_RSA_WITH_AES_256_GCM_SHA384

It appears we are seeing upgrade requests from servers through a TLS session, TLS1.1 requests to 1.2 or in some cases 1.3, this is not allowed per RFC. The firewall reads the request as a header frame, and then tries to extract sequential data from the incoming packets of the session. Since the session isn't terminated, the bogus packet(s) aren't dropped, session depletion happens. 

We are currently researching how to terminate the session when this type of request comes in out of order. 

[LCMember4169]

Yep, we just had to roll our HA environment back to 9.1.13-h3 because of this issue (RSA certs). Too bad it wasn't listed in the Known Issues.

[N2Z2 · Accepted solution]

It looks like that a workaround has been published or am I wrong?

PAN-194395

The firewall drops all decrypted outbound (SSL Forward Proxy) HTTP/2 traffic after you upgrade to PAN-OS 9.1.14. Dropping this traffic prevents users from loading HTTP/2 web pages and accessing websites that use HTTP/2.

Workaround: On the SSL Forward Proxy tab in the Decryption profile attached to the Decryption Policy rule that controls the HTTP/2 traffic, select Strip ALPN. When you Strip ALPN, the firewall negotiates HTTP/1.1 instead of HTTP/2.

It seems working in my environment.

[LAS]

i will wait then for when its fixed, as i don't want to downgrade HTTP/2 connections to HTTP/1.1 as a workaround

[LAYER_8]

Indeed, it is now documented in the known issues. If your security requirements allow for HTTP1.1 / strip ALPN workaround that will suffice. If not, we are still researching the TLS upgrade requests and out of order queue.  

[LAYER_8]

Root cause identified, the fix is coming in 9.1.15-h1. Your choices are upgrade (when available), strip ALPN (if allowed), downgrade/stay on 9.1.13-h3. Final answer 🙂 

[jpan123]

What is your TLS version? can you show the profile you are using this recommendation did not work. I will attached my profile here.

[N2Z2]

Hi,

Min 1.0, max version "Max"

The configuration is the same as in your screenshot