SSL Decryption bug in PAN-OS 9.1.14

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.
Palo Alto Networks Approved
Palo Alto Networks Approved
Community Expert Verified
Community Expert Verified

SSL Decryption bug in PAN-OS 9.1.14

L2 Linker

I recently upgraded from panos 9.1.13-h3 to 9.1.14 then SSL decryption stopped working, in the traffic monitor there wasn't any decryption error but when i excluded a PC the internet worked


and it seams other people are also having the same issue (Reddit ), but its not in the known issue list until now


so i had to revert to the previous version and its working now fine

1 accepted solution

Accepted Solutions

L2 Linker

It looks like that a workaround has been published or am I wrong?

 

PAN-194395
The firewall drops all decrypted outbound (SSL Forward Proxy) HTTP/2 traffic after you upgrade to PAN-OS 9.1.14. Dropping this traffic prevents users from loading HTTP/2 web pages and accessing websites that use HTTP/2.
Workaround: On the SSL Forward Proxy tab in the Decryption profile attached to the Decryption Policy rule that controls the HTTP/2 traffic, select Strip ALPN. When you Strip ALPN, the firewall negotiates HTTP/1.1 instead of HTTP/2.
 
It seems working in my environment.

View solution in original post

22 REPLIES 22

L5 Sessionator

I don't know if this is your exact issue, but it seems as if we are tracking something internally. PAN-194219, the software packet buffers are depleting erroneously during HTTP/2 inspection only post 9.1.14 upgrade, and subsequently not decrypting.  

 

I am now following the issue and will post updates / workarounds. 

Help the community! Add tags and mark solutions please.

Ok thank you, will not update until the issue is resolved

L0 Member

Same here. Rolling back now....I wonder why there is not any statement by PA. Or is it and I didn't find it? 

L5 Sessionator

The issue is escalated to the senior support queue, they've successfully recreated it in a lab and are now identifying root cause. 

Help the community! Add tags and mark solutions please.

L2 Linker

Hi,

could you confirm that the problem is present in both RSA and ECDSA algorithm?

There's a know issue (PAN-83215) for ECDSA but I got the error with RSA.

Thanks

L2 Linker

The certificate that i use is also RSA

L5 Sessionator

Good news! Root cause identified, yes, RSA will get stumped here, too:

 

NGFW> debug dataplane show ssl-decrypt session 321122

Session 321122(local 321122), 1.0.0.2[50393]-->2.0.0.2[443]
Proxy Flow
        Index: 721716, Type: proxy, Tag: 321122, Dir: cts
        Rule: CRPNY-Decrypt
        Profile: 18F-Outbound-Decrypt-Office
        4 Packets Pending for L7 Proc
        TCP state
                Server Established
                MSS 1460
                DELACK timer is not on, RXMT timer is not on
                Recv Next:  581584158, Window: 49152, Scale: 0
                Send Next:  590184458, Window:  5792, Scale: 0
                Send Max :  590184458, Send Unack:  590184458
                Slow Start Threshold: 1073725440
                Congestion Window: 17520, RTT: 0 ticks
                # of Out-of-Order Pkts: 0
                # of Retrans: 0, # of DupAcks: 0
                # of Unsent Pkts: 0, # of Unacked Pkts: 0
        SSL State
                Protocol Version: TLS1.2
                Cipher : TLS_RSA_WITH_AES_256_GCM_SHA384

Peer Flow
        Index: 721703, Type: proxy, Tag: 321122, Dir: stc
        Rule: CRPNY-Decrypt
        Profile: 18F-Outbound-Decrypt-Office
        Is Closed
        4 Packets Pending for L7 Proc
        TCP state
                Client Closed Wait
                Can't recv
                MSS 1460
                DELACK timer is not on, RXMT timer is not on
                Recv Next:  590184464, Window: 49152, Scale: 0
                Send Next:  581583672, Window:  5792, Scale: 0
                Send Max :  581583672, Send Unack:  581583672
                Slow Start Threshold: 1073725440
                Congestion Window: 17520, RTT: 0 ticks
                # of Out-of-Order Pkts: 0
                # of Retrans: 0, # of DupAcks: 0
                # of Unsent Pkts: 0, # of Unacked Pkts: 0
        SSL State
                Protocol Version: TLS1.2
                Cipher : TLS_RSA_WITH_AES_256_GCM_SHA384

It appears we are seeing upgrade requests from servers through a TLS session, TLS1.1 requests to 1.2 or in some cases 1.3, this is not allowed per RFC. The firewall reads the request as a header frame, and then tries to extract sequential data from the incoming packets of the session. Since the session isn't terminated, the bogus packet(s) aren't dropped, session depletion happens. 

 

We are currently researching how to terminate the session when this type of request comes in out of order. 

Help the community! Add tags and mark solutions please.

Yep, we just had to roll our HA environment back to 9.1.13-h3 because of this issue (RSA certs). Too bad it wasn't listed in the Known Issues.

L2 Linker

It looks like that a workaround has been published or am I wrong?

 

PAN-194395
The firewall drops all decrypted outbound (SSL Forward Proxy) HTTP/2 traffic after you upgrade to PAN-OS 9.1.14. Dropping this traffic prevents users from loading HTTP/2 web pages and accessing websites that use HTTP/2.
Workaround: On the SSL Forward Proxy tab in the Decryption profile attached to the Decryption Policy rule that controls the HTTP/2 traffic, select Strip ALPN. When you Strip ALPN, the firewall negotiates HTTP/1.1 instead of HTTP/2.
 
It seems working in my environment.

L2 Linker

i will wait then for when its fixed, as i don't want to downgrade HTTP/2 connections to HTTP/1.1 as a workaround

L5 Sessionator

Indeed, it is now documented in the known issues. If your security requirements allow for HTTP1.1 / strip ALPN workaround that will suffice. If not, we are still researching the TLS upgrade requests and out of order queue.  

Help the community! Add tags and mark solutions please.

L5 Sessionator

Root cause identified, the fix is coming in 9.1.15-h1. Your choices are upgrade (when available), strip ALPN (if allowed), downgrade/stay on 9.1.13-h3. Final answer 🙂 

Help the community! Add tags and mark solutions please.

What is your TLS version? can you show the profile you are using this recommendation did not work. I will attached my profile here.

 

jpan123_0-1655216655552.png

 

Hi,

Min 1.0, max version "Max"

The configuration is the same as in your screenshot

 

  • 1 accepted solution
  • 8031 Views
  • 22 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!