I feel like I'm probably missing something simple here, but I'm running into a decrypt-error issue on 8.1.3 in regards to a server that is negotiating DHE or ECDHE ciphers with the client. On Chrome I get:
And on Firefox I get:
If I turn off decryption for this or set it to RSA only, the traffic goes through (albeit not decrypted) so the client and server do indeed have shared ciphers they can negotiate to. A packet capture from my machine and an SSL Labs scan of the server seems to back this up.
I know DHE and ECDHE wasn't supported on Inbound Decryption before but the warning is no longer on the configuration GUI and I see newer documentation that it has been supported since an 8.0 version.
Any ideas on what I might be doing wrong or how I should proceed with troubleshooting?
The Firefox error is better in terms of what it's saying: no cipher overlap.
When your client sends the Client Hello, it will specify some number of cipher suites that it supports. When the server (your firewall in this case) gets that list, it tries to match with the highest security cipher in that list. But if the firewall doesn't have any that match the client's list, you will get the error that you got.
Check a few things:
1. Your decryption profile on the firewall should include at least one cipher that the client is sending. Go to Objects > Decryption > Decryption Profile and hit the SSL Protocol Settings on the profile you use in your decrypt rule for this traffic.
2. While you're there, make sure that the "Protocol Versions" is set with the max version of "Max". Chrome has recently stopped supporting TLS 1.0, and others (Firefox, Edge, Opera, Safari, etc.) may have as well or will be soon), so if you're maxing that out at TLS 1.0 or 1.1, you'll be out of luck for most browsers.
3. Make sure that your browsers are configured to send all the DHE and ECDHE ciphers it should, and that they haven't been artificially limited by some setting on your end.
4. Run a TLS check (Qualys has a popular one for free) to see what you're supporting if the site is publicly accessible.
Thanks for the reply.
This is as far as I understand it:
For decrypting inbound connections whereby RSA is used, the firewall doesn't have to proxy the connection since it has the private key, it can decrypt the traffic on the fly.
For DHE and ECDHE traffic, since there is a new public/private key pair for each session the firewall cannot simply decrypt the traffic as it passes through, it instead needs to proxy with the server. Perhaps if you do a packet capture on the server side of things you should see traffic from there to the firewall with the SSL handshake failing?
Thanks for providing the screenshots!
With that data, it looks like it should work without issue.
Decryption was enabled when you did the Qualys scan right?If no, you'll need to re-enable it and re-run it to ensure it still matches. If decryption was enabled when you ran it, my next recommendation would actually be to open a support case unless you're comfortable posting the details here.
We went ahead and did a packet capture on the server. I'm not seeing any communication between it and the firewall.
I see the following, all between my desktop and the server:
I only have decryption enabled between my client and the server right now. The scan from SSL Labs would have not gone through any decryption process on the firewall. I wasn't sure if the scan would work since my browsers don't seem to get very far into the process when it is enabled and I didn't want to break anything for Internet traffic hitting the server.
I'll see if I can find an IP range they use so I can enable it just for that traffic.
> I'm not seeing any communication between it and the firewall.
If you've got decryption enabled between your desktop and the server, then those packets should be the firewall. The IP will still be the client's public IP unless you're doing source NAT for that traffic to the server. I filled in what the bits I think are most relevant to the flow:
Client Hello -->
<-- Server Hello (+cert, etc.)
Most likely that is actually from the firewall, and the RST you're seeing is because the firewall is denying the traffic based on Server Hello. The most common reasons I see when it happens at the Server Hello are a URL category block or an aggressive narrowly-defined application policy.
You can also check your session on the firewall to see the state when that RST happens. Assuming the client IP is 192.0.2.1, trigger the reset then in CLI:
> show session all filter source 192.0.2.1
Once you see the session that matches your client it should be in DISCARD state. Then, take the session ID and do:
> show session id 12345678
That should show you more details for the reason.
Here is my session output:
source: desktop IP [CampusCore]
dst: server IP
sport: 64022 dport: 443
state: DISCARD type: FLOW
src user: jsalmans
dst user: unknown
ecmp id: 8001
source: server IP [DataCenter]
dst: desktop IP
sport: 443 dport: 64022
state: DISCARD type: FLOW
src user: unknown
dst user: jsalmans
Slot : 1
DP : 1
index(local): : 1879165
start time : Tue Feb 5 12:03:56 2019
timeout : 90 sec
time to live : 28 sec
total byte count(c2s) : 1797
total byte count(s2c) : 5445
layer7 packet count(c2s) : 20
layer7 packet count(s2c) : 7
vsys : vsys1
application : ssl
rule : Allow Dept Anywhere
service timeout override(index) : False
session to be logged at end : True
session in session ager : True
session updated by HA peer : False
layer7 processing : enabled
URL filtering enabled : True
URL category : educational-institutions
session via syn-cookies : False
session terminated on host : False
session traverses tunnel : False
captive portal session : False
ingress interface : ae1.10
egress interface : ae1.4001
session QoS rule : N/A (class 4)
tracker stage firewall : proxy decrypt failure
end-reason : decrypt-error
The chain warning just means that the server (firewall in this case) isn't sending the intermediate CAs. It's not usually a problem and will not cause the issue you're seeing, but also has a way to totally eliminate it. Here's an article I wrote that goes into the details:
As for the handshake_failure warnings seen, those are the real problem. That the Qualys scan shows it failing Chrome 69 and 70 echoes what you see.
I can tell you that the firewall is the one causing this, but whether it's configuration or something else would probably need to be investigated further using the firewall logs and some debugs. I won't suggest that here, it's too dangerous to do unless you're very familiar with the debugging process for proxy-based decryption.
I would recommend opening a support ticket, and would suggest pointing the engineer who gets your ticket to this thread as well.
@gwessonnot sure how I missed that with the Intermediary cert but thanks! I had it uploaded for on Panorama but not in the Device Group that pushed to the firewall... I added it and it immediately changed the Wildcard to be a sub-cert. I did another scan and it hows correct now.
I'm opening a TAC case for the SSL decryption issue and I'll reply again when I have a solution in case someone else runs into this as well.
So an update.. it was determined the server and client are trying to use X25519 which is an ECDHE curve that Palo Alto doesn't support (definitely would be nice to see this as a note on the supported ciphers page... TLSv1.3 uses it as a standard and I know that isn't supported yet but TLSv1.2 uses it as well).
The workaround is to disable ECDHE but that doesn't seem like a great call given that we're talking about lowering server security to apply SSL Decryption for additional server security.
I found this for Windows Server 2016 and it seems to work:
I'm looking for similar instructions for Apache and Tomcat. I'm not a server expert and I'm having trouble finding methods to do this on those platforms.
With my (limited) Apache knowledge, you don't strictly exclude specific curves but rather include only the ones you want. You would put a line in your httpd.conf (or apache.conf, or whatever your site uses). It will probably wrap when I post this, but it will all be on one line. I stole this from Apache Lounge:
SSLOpenSSLConfCmd Curves sect571r1:sect571k1:secp521r1:sect409k1:sect409r1:secp384r1:sect283k1:sect2...
You can also use the SSLCipherSuite directive to exclude entire suites as needed.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!