Decrypt-error with Inbound Decryption DHE or ECDHE on 8.1.3

[jsalmans]

Greetings all,

I feel like I'm probably missing something simple here, but I'm running into a decrypt-error issue on 8.1.3 in regards to a server that is negotiating DHE or ECDHE ciphers with the client.  On Chrome I get:

ERR_SSL_VERSION_OR_CIPHER_MISMATCH

And on Firefox I get:

SSL_ERROR_NO_CYPHER_OVERLAP

If I turn off decryption for this or set it to RSA only, the traffic goes through (albeit not decrypted) so the client and server do indeed have shared ciphers they can negotiate to.  A packet capture from my machine and an SSL Labs scan of the server seems to back this up.

I know DHE and ECDHE wasn't supported on Inbound Decryption before but the warning is no longer on the configuration GUI and I see newer documentation that it has been supported since an 8.0 version.

Any ideas on what I might be doing wrong or how I should proceed with troubleshooting?

[gwesson]

The Firefox error is better in terms of what it's saying: no cipher overlap.

When your client sends the Client Hello, it will specify some number of cipher suites that it supports. When the server (your firewall in this case) gets that list, it tries to match with the highest security cipher in that list. But if the firewall doesn't have any that match the client's list, you will get the error that you got.

Check a few things:

1. Your decryption profile on the firewall should include at least one cipher that the client is sending. Go to Objects > Decryption > Decryption Profile and hit the SSL Protocol Settings on the profile you use in your decrypt rule for this traffic.

2. While you're there, make sure that the "Protocol Versions" is set with the max version of "Max". Chrome has recently stopped supporting TLS 1.0, and others (Firefox, Edge, Opera, Safari, etc.) may have as well or will be soon), so if you're maxing that out at TLS 1.0 or 1.1, you'll be out of luck for most browsers.

3. Make sure that your browsers are configured to send all the DHE and ECDHE ciphers it should, and that they haven't been artificially limited by some setting on your end.

4. Run a TLS check (Qualys has a popular one for free) to see what you're supporting if the site is publicly accessible.

[MP18]

Very good Explanation.

[jsalmans]

Thanks for the reply.

1. I believe I've done that. 
   
   
   
   
2. I've tried this as Max and also setting TLS 1.2 as the max.
3. As far as I know, Firefox and Chrome are both using whatever the defaults are.  If I turn off decryption, I can see it negotiating successfully with the server wiht an ECDHS cipher.  Here is what the packet capture is seeing my client offer.
   
   
   
   
4. Here is the SSL Labs results for TLS 1.2:
   
   

[LukeBullimore]

This is as far as I understand it:

For decrypting inbound connections whereby RSA is used, the firewall doesn't have to proxy the connection since it has the private key, it can decrypt the traffic on the fly.

For DHE and ECDHE traffic, since there is a new public/private key pair for each session the firewall cannot simply decrypt the traffic as it passes through, it instead needs to proxy with the server. Perhaps if you do a packet capture on the server side of things you should see traffic from there to the firewall with the SSL handshake failing?

Cheers,

Luke.

[gwesson]

Thanks for providing the screenshots!

With that data, it looks like it should work without issue.

Decryption was enabled when you did the Qualys scan right?If no, you'll need to re-enable it and re-run it to ensure it still matches. If decryption was enabled when you ran it, my next recommendation would actually be to open a support case unless you're comfortable posting the details here.

[jsalmans]

@LukeBullimore

We went ahead and did a packet capture on the server.  I'm not seeing any communication between it and the firewall.

I see the following, all between my desktop and the server:

• Initial SYN from my desktop
• ACK from server
• Client Hello on TLSv1.2 from my desktop
• Server Hello, Certificate, Certificate Status, Server Key Exchange, Server Hello Done from server
• RST, ACK from my desktop
• ACK from my desktop
• ACK from my desktop
• TCP Retransmission from my desktop
• ACK from server
• RST, ACK from desktop
• RST, ACK from server

[jsalmans]

@gwesson

I only have decryption enabled between my client and the server right now.  The scan from SSL Labs would have not gone through any decryption process on the firewall.  I wasn't sure if the scan would work since my browsers don't seem to get very far into the process when it is enabled and I didn't want to break anything for Internet traffic hitting the server.

I'll see if I can find an IP range they use so I can enable it just for that traffic.

[gwesson]

> I'm not seeing any communication between it and the firewall.

If you've got decryption enabled between your desktop and the server, then those packets should be the firewall. The IP will still be the client's public IP unless you're doing source NAT for that traffic to the server. I filled in what the bits I think are most relevant to the flow:

    SYN -->

<-- SYN,ACK

    ACK -->

    Client Hello --> 

<-- ACK

<-- Server Hello (+cert, etc.)

    RST/ACK -->

Most likely that is actually from the firewall, and the RST you're seeing is because the firewall is denying the traffic based on Server Hello. The most common reasons I see when it happens at the Server Hello are a URL category block or an aggressive narrowly-defined application policy. 

You can also check your session on the firewall to see the state when that RST happens. Assuming the client IP is 192.0.2.1, trigger the reset then in CLI:

> show session all filter source 192.0.2.1

Once you see the session that matches your client it should be in DISCARD state. Then, take the session ID and do:

> show session id 12345678

That should show you more details for the reason.

[jsalmans]

@gwesson

Here is my session output:

Session        35433597

        c2s flow:
                source:      desktop IP [CampusCore]
                dst:         server IP
                proto:       6
                sport:       64022           dport:      443
                state:       DISCARD         type:       FLOW
                src user:   jsalmans
                dst user:    unknown
                offload:     Yes
                ecmp id:     8001

        s2c flow:
                source:      server IP [DataCenter]
                dst:         desktop IP
                proto:       6
                sport:       443             dport:      64022
                state:       DISCARD         type:       FLOW
                src user:    unknown
                dst user:    jsalmans
                offload:     Yes

        Slot                                 : 1
        DP                                   : 1
        index(local):                        : 1879165
        start time                           : Tue Feb  5 12:03:56 2019
        timeout                              : 90 sec
        time to live                         : 28 sec
        total byte count(c2s)                : 1797
        total byte count(s2c)                : 5445
        layer7 packet count(c2s)             : 20
        layer7 packet count(s2c)             : 7
        vsys                                 : vsys1
        application                          : ssl
        rule                                 : Allow Dept Anywhere
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : True
        session updated by HA peer           : False
        layer7 processing                    : enabled
        URL filtering enabled                : True
        URL category                         : educational-institutions
        session via syn-cookies              : False
        session terminated on host           : False
        session traverses tunnel             : False
        captive portal session               : False
        ingress interface                    : ae1.10
        egress interface                     : ae1.4001
        session QoS rule                     : N/A (class 4)
        tracker stage firewall               : proxy decrypt failure
        end-reason                           : decrypt-error