topic Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3 in General Topics

Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Mon, 04 Feb 2019 18:31:47 GMT

Greetings all,

I feel like I'm probably missing something simple here, but I'm running into a decrypt-error issue on 8.1.3 in regards to a server that is negotiating DHE or ECDHE ciphers with the client. On Chrome I get:

ERR_SSL_VERSION_OR_CIPHER_MISMATCH

And on Firefox I get:

SSL_ERROR_NO_CYPHER_OVERLAP

If I turn off decryption for this or set it to RSA only, the traffic goes through (albeit not decrypted) so the client and server do indeed have shared ciphers they can negotiate to. A packet capture from my machine and an SSL Labs scan of the server seems to back this up.

I know DHE and ECDHE wasn't supported on Inbound Decryption before but the warning is no longer on the configuration GUI and I see newer documentation that it has been supported since an 8.0 version.

Any ideas on what I might be doing wrong or how I should proceed with troubleshooting?

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

gwesson — Mon, 04 Feb 2019 18:58:27 GMT

The Firefox error is better in terms of what it's saying: no cipher overlap.

When your client sends the Client Hello, it will specify some number of cipher suites that it supports. When the server (your firewall in this case) gets that list, it tries to match with the highest security cipher in that list. But if the firewall doesn't have any that match the client's list, you will get the error that you got.

Check a few things:

1. Your decryption profile on the firewall should include at least one cipher that the client is sending. Go to Objects > Decryption > Decryption Profile and hit the SSL Protocol Settings on the profile you use in your decrypt rule for this traffic.

2. While you're there, make sure that the "Protocol Versions" is set with the max version of "Max". Chrome has recently stopped supporting TLS 1.0, and others (Firefox, Edge, Opera, Safari, etc.) may have as well or will be soon), so if you're maxing that out at TLS 1.0 or 1.1, you'll be out of luck for most browsers.

3. Make sure that your browsers are configured to send all the DHE and ECDHE ciphers it should, and that they haven't been artificially limited by some setting on your end.

4. Run a TLS check (Qualys has a popular one for free) to see what you're supporting if the site is publicly accessible.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

MP18 — Mon, 04 Feb 2019 20:16:32 GMT

Very good Explanation.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Tue, 05 Feb 2019 02:59:42 GMT

Thanks for the reply.

I believe I've done that.
I've tried this as Max and also setting TLS 1.2 as the max.
As far as I know, Firefox and Chrome are both using whatever the defaults are. If I turn off decryption, I can see it negotiating successfully with the server wiht an ECDHS cipher. Here is what the packet capture is seeing my client offer.
Here is the SSL Labs results for TLS 1.2:

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

LukeBullimore — Tue, 05 Feb 2019 13:43:12 GMT

This is as far as I understand it:

For decrypting inbound connections whereby RSA is used, the firewall doesn't have to proxy the connection since it has the private key, it can decrypt the traffic on the fly.

For DHE and ECDHE traffic, since there is a new public/private key pair for each session the firewall cannot simply decrypt the traffic as it passes through, it instead needs to proxy with the server. Perhaps if you do a packet capture on the server side of things you should see traffic from there to the firewall with the SSL handshake failing?

Cheers,

Luke.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

gwesson — Tue, 05 Feb 2019 17:39:45 GMT

Thanks for providing the screenshots!

With that data, it looks like it should work without issue.

Decryption was enabled when you did the Qualys scan right?If no, you'll need to re-enable it and re-run it to ensure it still matches. If decryption was enabled when you ran it, my next recommendation would actually be to open a support case unless you're comfortable posting the details here.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Tue, 05 Feb 2019 17:41:24 GMT

@LukeBullimore

We went ahead and did a packet capture on the server. I'm not seeing any communication between it and the firewall.

I see the following, all between my desktop and the server:

Initial SYN from my desktop
ACK from server
Client Hello on TLSv1.2 from my desktop
Server Hello, Certificate, Certificate Status, Server Key Exchange, Server Hello Done from server
RST, ACK from my desktop
ACK from my desktop
ACK from my desktop
TCP Retransmission from my desktop
ACK from server
RST, ACK from desktop
RST, ACK from server

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Tue, 05 Feb 2019 17:43:42 GMT

@gwesson

I only have decryption enabled between my client and the server right now. The scan from SSL Labs would have not gone through any decryption process on the firewall. I wasn't sure if the scan would work since my browsers don't seem to get very far into the process when it is enabled and I didn't want to break anything for Internet traffic hitting the server.

I'll see if I can find an IP range they use so I can enable it just for that traffic.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

gwesson — Tue, 05 Feb 2019 18:00:05 GMT

> I'm not seeing any communication between it and the firewall.

If you've got decryption enabled between your desktop and the server, then those packets should be the firewall. The IP will still be the client's public IP unless you're doing source NAT for that traffic to the server. I filled in what the bits I think are most relevant to the flow:

SYN -->

<-- SYN,ACK

ACK -->

Client Hello -->

<-- ACK

<-- Server Hello (+cert, etc.)

RST/ACK -->

Most likely that is actually from the firewall, and the RST you're seeing is because the firewall is denying the traffic based on Server Hello. The most common reasons I see when it happens at the Server Hello are a URL category block or an aggressive narrowly-defined application policy.

You can also check your session on the firewall to see the state when that RST happens. Assuming the client IP is 192.0.2.1, trigger the reset then in CLI:

> show session all filter source 192.0.2.1

Once you see the session that matches your client it should be in DISCARD state. Then, take the session ID and do:

> show session id 12345678

That should show you more details for the reason.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Tue, 05 Feb 2019 18:08:21 GMT

@gwesson

Here is my session output:

Session        35433597

        c2s flow:
                source:      desktop IP [CampusCore]
                dst:         server IP
                proto:       6
                sport:       64022           dport:      443
                state:       DISCARD         type:       FLOW
                src user:   jsalmans
                dst user:    unknown
                offload:     Yes
                ecmp id:     8001

        s2c flow:
                source:      server IP [DataCenter]
                dst:         desktop IP
                proto:       6
                sport:       443             dport:      64022
                state:       DISCARD         type:       FLOW
                src user:    unknown
                dst user:    jsalmans
                offload:     Yes

        Slot                                 : 1
        DP                                   : 1
        index(local):                        : 1879165
        start time                           : Tue Feb 5 12:03:56 2019
        timeout                              : 90 sec
        time to live                         : 28 sec
        total byte count(c2s)                : 1797
        total byte count(s2c)                : 5445
        layer7 packet count(c2s)             : 20
        layer7 packet count(s2c)             : 7
        vsys                                 : vsys1
        application                          : ssl
        rule                                 : Allow Dept Anywhere
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : True
        session updated by HA peer           : False
        layer7 processing                    : enabled
        URL filtering enabled                : True
        URL category                         : educational-institutions
        session via syn-cookies              : False
        session terminated on host           : False
        session traverses tunnel             : False
        captive portal session               : False
        ingress interface                    : ae1.10
        egress interface                     : ae1.4001
        session QoS rule                     : N/A (class 4)
        tracker stage firewall               : proxy decrypt failure
        end-reason                           : decrypt-error

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Tue, 05 Feb 2019 18:10:52 GMT

I found the SSL Labs IP range. Some of the traffic does appear to be decrypted while some of it hits a decrypt-error.

I also see "This server's certificate chain is incomplete. Grade capped to B. "

Here is the current decryption profile for reference

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

gwesson — Tue, 05 Feb 2019 18:26:14 GMT

The chain warning just means that the server (firewall in this case) isn't sending the intermediate CAs. It's not usually a problem and will not cause the issue you're seeing, but also has a way to totally eliminate it. Here's an article I wrote that goes into the details:

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClkoCAC

As for the handshake_failure warnings seen, those are the real problem. That the Qualys scan shows it failing Chrome 69 and 70 echoes what you see.

I can tell you that the firewall is the one causing this, but whether it's configuration or something else would probably need to be investigated further using the firewall logs and some debugs. I won't suggest that here, it's too dangerous to do unless you're very familiar with the debugging process for proxy-based decryption.

I would recommend opening a support ticket, and would suggest pointing the engineer who gets your ticket to this thread as well.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Tue, 05 Feb 2019 18:59:10 GMT

@gwessonnot sure how I missed that with the Intermediary cert but thanks! I had it uploaded for on Panorama but not in the Device Group that pushed to the firewall... I added it and it immediately changed the Wildcard to be a sub-cert. I did another scan and it hows correct now.

I'm opening a TAC case for the SSL decryption issue and I'll reply again when I have a solution in case someone else runs into this as well.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

jsalmans — Wed, 13 Feb 2019 16:24:11 GMT

So an update.. it was determined the server and client are trying to use X25519 which is an ECDHE curve that Palo Alto doesn't support (definitely would be nice to see this as a note on the supported ciphers page... TLSv1.3 uses it as a standard and I know that isn't supported yet but TLSv1.2 uses it as well).

The workaround is to disable ECDHE but that doesn't seem like a great call given that we're talking about lowering server security to apply SSL Decryption for additional server security.

I found this for Windows Server 2016 and it seems to work:

https://www.nsgp.net/2018/09/how-to-disable-curve25519-x25519-key-exchange-on-windows-server-2016/

I'm looking for similar instructions for Apache and Tomcat. I'm not a server expert and I'm having trouble finding methods to do this on those platforms.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

gwesson — Wed, 13 Feb 2019 18:47:29 GMT

With my (limited) Apache knowledge, you don't strictly exclude specific curves but rather include only the ones you want. You would put a line in your httpd.conf (or apache.conf, or whatever your site uses). It will probably wrap when I post this, but it will all be on one line. I stole this from Apache Lounge:

SSLOpenSSLConfCmd Curves sect571r1:sect571k1:secp521r1:sect409k1:sect409r1:secp384r1:sect283k1:sect2...

You can also use the SSLCipherSuite directive to exclude entire suites as needed.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

benjamins — Wed, 22 May 2019 17:02:28 GMT

Did you get any further? I am struggling with similar issue and have been debugging for two days now getting further and further. In case of Apache I can say there seems to be a lot of requirements to get decryption working on latest version 2.4.39 (which i haven't managed to get fully working yet)

Benjamin

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

smc007 — Mon, 02 Dec 2019 18:07:21 GMT

Very disappointing that I cannot use Palo for SSL inspection any more due to our load balancer AVI, is using OpenSSL v1.1.1 which defaults to X25519 curve for ECDEH and DHE PFS ciphers using TLS1.2/1.1 so the only option I have is to use the RSA-AES-256-GCM or CBC ciphers which then caps you SSL security rating to B vs A+ with ECDHE ciphers.

Come on Palo, fix the issue and support the curve x25519, you need that curve for TLS1.3 support.

Re: Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

CLIq — Mon, 24 May 2021 20:02:56 GMT

You will need to feature request that...
For Apache and nginx this requires a specific configuration and order (actually x25519 does not seem to be needed for tls1.3 but rsa_pss signature algorithm is and that is also not supported for tls1.2) to work... but it does work!
So if you do feature request this with your SE, make sure you also request support for rsa_pss signature algorithm 🙂