Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

[jsalmans]

Greetings all,

I feel like I'm probably missing something simple here, but I'm running into a decrypt-error issue on 8.1.3 in regards to a server that is negotiating DHE or ECDHE ciphers with the client.  On Chrome I get:

ERR_SSL_VERSION_OR_CIPHER_MISMATCH

And on Firefox I get:

SSL_ERROR_NO_CYPHER_OVERLAP

If I turn off decryption for this or set it to RSA only, the traffic goes through (albeit not decrypted) so the client and server do indeed have shared ciphers they can negotiate to.  A packet capture from my machine and an SSL Labs scan of the server seems to back this up.

I know DHE and ECDHE wasn't supported on Inbound Decryption before but the warning is no longer on the configuration GUI and I see newer documentation that it has been supported since an 8.0 version.

Any ideas on what I might be doing wrong or how I should proceed with troubleshooting?

[gwesson]

The Firefox error is better in terms of what it's saying: no cipher overlap.

When your client sends the Client Hello, it will specify some number of cipher suites that it supports. When the server (your firewall in this case) gets that list, it tries to match with the highest security cipher in that list. But if the firewall doesn't have any that match the client's list, you will get the error that you got.

Check a few things:

1. Your decryption profile on the firewall should include at least one cipher that the client is sending. Go to Objects > Decryption > Decryption Profile and hit the SSL Protocol Settings on the profile you use in your decrypt rule for this traffic.

2. While you're there, make sure that the "Protocol Versions" is set with the max version of "Max". Chrome has recently stopped supporting TLS 1.0, and others (Firefox, Edge, Opera, Safari, etc.) may have as well or will be soon), so if you're maxing that out at TLS 1.0 or 1.1, you'll be out of luck for most browsers.

3. Make sure that your browsers are configured to send all the DHE and ECDHE ciphers it should, and that they haven't been artificially limited by some setting on your end.

4. Run a TLS check (Qualys has a popular one for free) to see what you're supporting if the site is publicly accessible.

[MP18]

Very good Explanation.

[jsalmans]

Thanks for the reply.

1. I believe I've done that. 
   
   
   
   
2. I've tried this as Max and also setting TLS 1.2 as the max.
3. As far as I know, Firefox and Chrome are both using whatever the defaults are.  If I turn off decryption, I can see it negotiating successfully with the server wiht an ECDHS cipher.  Here is what the packet capture is seeing my client offer.
   
   
   
   
4. Here is the SSL Labs results for TLS 1.2:
   
   

[LukeBullimore]

This is as far as I understand it:

For decrypting inbound connections whereby RSA is used, the firewall doesn't have to proxy the connection since it has the private key, it can decrypt the traffic on the fly.

For DHE and ECDHE traffic, since there is a new public/private key pair for each session the firewall cannot simply decrypt the traffic as it passes through, it instead needs to proxy with the server. Perhaps if you do a packet capture on the server side of things you should see traffic from there to the firewall with the SSL handshake failing?

Cheers,

Luke.

[gwesson]

Thanks for providing the screenshots!

With that data, it looks like it should work without issue.

Decryption was enabled when you did the Qualys scan right?If no, you'll need to re-enable it and re-run it to ensure it still matches. If decryption was enabled when you ran it, my next recommendation would actually be to open a support case unless you're comfortable posting the details here.

[jsalmans]

@LukeBullimore

We went ahead and did a packet capture on the server.  I'm not seeing any communication between it and the firewall.

I see the following, all between my desktop and the server:

• Initial SYN from my desktop
• ACK from server
• Client Hello on TLSv1.2 from my desktop
• Server Hello, Certificate, Certificate Status, Server Key Exchange, Server Hello Done from server
• RST, ACK from my desktop
• ACK from my desktop
• ACK from my desktop
• TCP Retransmission from my desktop
• ACK from server
• RST, ACK from desktop
• RST, ACK from server

[jsalmans]

@gwesson

I only have decryption enabled between my client and the server right now.  The scan from SSL Labs would have not gone through any decryption process on the firewall.  I wasn't sure if the scan would work since my browsers don't seem to get very far into the process when it is enabled and I didn't want to break anything for Internet traffic hitting the server.

I'll see if I can find an IP range they use so I can enable it just for that traffic.

[gwesson]

> I'm not seeing any communication between it and the firewall.

If you've got decryption enabled between your desktop and the server, then those packets should be the firewall. The IP will still be the client's public IP unless you're doing source NAT for that traffic to the server. I filled in what the bits I think are most relevant to the flow:

    SYN -->

<-- SYN,ACK

    ACK -->

    Client Hello --> 

<-- ACK

<-- Server Hello (+cert, etc.)

    RST/ACK -->

Most likely that is actually from the firewall, and the RST you're seeing is because the firewall is denying the traffic based on Server Hello. The most common reasons I see when it happens at the Server Hello are a URL category block or an aggressive narrowly-defined application policy. 

You can also check your session on the firewall to see the state when that RST happens. Assuming the client IP is 192.0.2.1, trigger the reset then in CLI:

> show session all filter source 192.0.2.1

Once you see the session that matches your client it should be in DISCARD state. Then, take the session ID and do:

> show session id 12345678

That should show you more details for the reason.

[jsalmans]

@gwesson

Here is my session output:

Session        35433597

        c2s flow:
                source:      desktop IP [CampusCore]
                dst:         server IP
                proto:       6
                sport:       64022           dport:      443
                state:       DISCARD         type:       FLOW
                src user:   jsalmans
                dst user:    unknown
                offload:     Yes
                ecmp id:     8001

        s2c flow:
                source:      server IP [DataCenter]
                dst:         desktop IP
                proto:       6
                sport:       443             dport:      64022
                state:       DISCARD         type:       FLOW
                src user:    unknown
                dst user:    jsalmans
                offload:     Yes

        Slot                                 : 1
        DP                                   : 1
        index(local):                        : 1879165
        start time                           : Tue Feb  5 12:03:56 2019
        timeout                              : 90 sec
        time to live                         : 28 sec
        total byte count(c2s)                : 1797
        total byte count(s2c)                : 5445
        layer7 packet count(c2s)             : 20
        layer7 packet count(s2c)             : 7
        vsys                                 : vsys1
        application                          : ssl
        rule                                 : Allow Dept Anywhere
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : True
        session updated by HA peer           : False
        layer7 processing                    : enabled
        URL filtering enabled                : True
        URL category                         : educational-institutions
        session via syn-cookies              : False
        session terminated on host           : False
        session traverses tunnel             : False
        captive portal session               : False
        ingress interface                    : ae1.10
        egress interface                     : ae1.4001
        session QoS rule                     : N/A (class 4)
        tracker stage firewall               : proxy decrypt failure
        end-reason                           : decrypt-error

[jsalmans]

I found the SSL Labs IP range.  Some of the traffic does appear to be decrypted while some of it hits a decrypt-error.

I also see "This server's certificate chain is incomplete. Grade capped to B. "

Here is the current decryption profile for reference

[gwesson]

The chain warning just means that the server (firewall in this case) isn't sending the intermediate CAs. It's not usually a problem and will not cause the issue you're seeing, but also has a way to totally eliminate it. Here's an article I wrote that goes into the details:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClkoCAC

As for the handshake_failure warnings seen, those are the real problem. That the Qualys scan shows it failing Chrome 69 and 70 echoes what you see. 

I can tell you that the firewall is the one causing this, but whether it's configuration or something else would probably need to be investigated further using the firewall logs and some debugs. I won't suggest that here, it's too dangerous to do unless you're very familiar with the debugging process for proxy-based decryption. 

I would recommend opening a support ticket, and would suggest pointing the engineer who gets your ticket to this thread as well.

[jsalmans]

@gwessonnot sure how I missed that with the Intermediary cert but thanks!  I had it uploaded for on Panorama but not in the Device Group that pushed to the firewall... I added it and it immediately changed the Wildcard to be a sub-cert.  I did another scan and it hows correct now.

I'm opening a TAC case for the SSL decryption issue and I'll reply again when I have a solution in case someone else runs into this as well.

[jsalmans]

So an update.. it was determined the server and client are trying to use X25519 which is an ECDHE curve that Palo Alto doesn't support (definitely would be nice to see this as a note on the supported ciphers page... TLSv1.3 uses it as a standard and I know that isn't supported yet but TLSv1.2 uses it as well).

The workaround is to disable ECDHE but that doesn't seem like a great call given that we're talking about lowering server security to apply SSL Decryption for additional server security.

I found this for Windows Server 2016 and it seems to work:

https://www.nsgp.net/2018/09/how-to-disable-curve25519-x25519-key-exchange-on-windows-server-2016/

I'm looking for similar instructions for Apache and Tomcat.  I'm not a server expert and I'm having trouble finding methods to do this on those platforms.

[gwesson]

With my (limited) Apache knowledge, you don't strictly exclude specific curves but rather include only the ones you want. You would put a line in your httpd.conf (or apache.conf, or whatever your site uses). It will probably wrap when I post this, but it will all be on one line. I stole this from Apache Lounge:

SSLOpenSSLConfCmd Curves sect571r1:sect571k1:secp521r1:sect409k1:sect409r1:secp384r1:sect283k1:sect2... 

You can also use the SSLCipherSuite directive to exclude entire suites as needed.