topic Re: Inbound SSL Decryption issues in General Topics

Inbound SSL Decryption issues

jsalmans — Tue, 01 Aug 2017 03:34:41 GMT

Wondering if anyone has an idea on why I might be getting "decrypt-error" on an Inbound SSL decrypt rule?

This service only runs a few times at night so I haven't done a packet capture yet... tonight I did some debug commands and found this in the log:

2017-07-31 22:00:15.865 -0500 Error: pan_ssl3_client_process_handshake(pan_ssl_client.c:871): pan_ssl3_client_get_se
rver_hello() failed
2017-07-31 22:00:15.865 -0500 Error: pan_ssl_proxy_handle_rt_hs(pan_ssl_proxy.c:236): pan_ssl3_process_handshake_mes
sage() failed -6
2017-07-31 22:00:15.865 -0500 Error: pan_ssl_proxy_parse_data(pan_ssl_proxy.c:550): pan_ssl_parse_record() failed

I didn't think this service was supposed to accept SSLv3 and my decryption profile I have installed shouldn't have that as an option. Even when I had no decryption profile it was giving me decrypt-error although I believe I was also seeing some decrypt-unsupport-param.

I'm on PANOS 7.1.11.

Thanks

Re: Inbound SSL Decryption issues

Remo — Tue, 01 Aug 2017 08:11:16 GMT

Hi @jsalmans

Is it working only from some clients or is this issue oersistent for all connections?

A packet capure will probably the best right now. With the capture you'll be able to see what ciphers the client proposes in the ssl-handshake hello packet. With that information it will probably already clear why the decryption isn't working.

Re: Inbound SSL Decryption issues

jsalmans — Tue, 01 Aug 2017 15:11:55 GMT

Hi @Remo

This is Indound Decryption on a security rule that only allows a single company to connect to an inside resource on our network. They do have a small list of IP addresses that can make the connection and, so far, I haven't seen it succeed on any of them. I verified with the vendor this morning that SSLv3 should be disabled and it shouldn't be using it... it should be using TLS1.2.

I'll do a packet capture at the next available connection which should be later tonight and see what I get.

Re: Inbound SSL Decryption issues

jsalmans — Wed, 02 Aug 2017 03:46:35 GMT

@Remo

So I'm looking at the packet capture but I'm not seeing any SSL transations or Cipher setup. I'm fairly certain I captured all of the traffic but the wireshark file is showing nothing but TCP protocol traffic and that includes the SYN, ACK from the server, some data PSH transfers, and then a RST from the server.

I'm fairly certain this data is supposed to be encrypted and the firewall is definitely showing it as SSL traffic and not as the application it should be coming across as.

This is one of my first attempts at Inbound SSL Decryption and its pretty late at night so hopefully I'm not making an obvious error here somewhere.

Re: Inbound SSL Decryption issues

Remo — Wed, 02 Aug 2017 17:07:45 GMT

@jsalmans

There definately should be more in the capture ...

What if you temporary allow the access from everywhere and run a TLS test like Qualys SSLLabs ot the one at htbridge.com?

Or you do a connection test yourself over the internet to check if there is a reason for this problem?

Or you share some screenshots of your security/decryption policy... may be there is something real simple thats missing?

Re: Inbound SSL Decryption issues

jsalmans — Wed, 02 Aug 2017 19:10:54 GMT

Hi @Remo

Yeah I'm not sure what was going on. I'm going to try again during the scheduled time tonight.

I thought about using one of those 3rd party SSL check services. I still might be it isn't my server and I'm not 100% responsible for the data it has so I'll have to reach out to my team on that.

Re: Inbound SSL Decryption issues

jsalmans — Thu, 03 Aug 2017 03:37:14 GMT

@Remo

So I did the packet capture again tonight. My filter was set up to capture any traffic going to the NAT IP or private IP of the server using the interface we have configured for Internet/Untrusted as the Ingress. I also had rules to capture any traffic coming from the NAT IP or the private IP. I also turned the decryption policy completely off for now. Here are the logs I see.

As you can see, app-id is flagging that as SSL traffic. I have my security policy set up like so:

Rule 75 was the original until I saw that the traffic was coming in as SSL and I created rule 74 to allow that traffic in (I'm assuming this won't be necessary once decryption is working since the decryption is supposed to happen before app-id according to the flow chart).

Here is a sample of what I'm seeing in Wireshark. I do see TLS traffic happening but between the server and a completely different client... when I limit my packet capture to those 52.x.x.x addresses I don't see anything except TCP.

Also, spyware alerts are being flagged from the 52.x.x.x IP address as "Suspicious TLC Evasion Found" but it is set to alert only.

Am I missing something really obvious here or am I seeing something like a false positive on ssl app-id?

Re: Inbound SSL Decryption issues

jsalmans — Tue, 08 Aug 2017 20:05:23 GMT

Just checking to see if anyone has any further ideas based on the info and screenshots I provided?

Thanks!

Re: Inbound SSL Decryption issues

gwesson — Tue, 08 Aug 2017 20:19:01 GMT

> when I limit my packet capture to those 52.x.x.x addresses I don't see anything except TCP.

In Wireshark, right-click on any of the frames you put in the capture > "Decode As" and choose SSL (change the port to the one you're using, 5671. Don't bother with the source port.):

This will let Wireshark display the actual Client Hello and certificate and such. By default, it only recognizes port 443 and a few others as SSL/TLS, so you have to tell it what dissector to use when it's not that set of ports.

Once you've got that in, you should be able to see where the failure is. Most of the time I see handshake failures caused by something simple like cipher suites not matching. It could be that there's something else, but the capture should help once it's decoded.

As for your original question regarding SSLv3, that's just the name in the code. Something like "pan_ssl3_process_handshake_message()" is just a function name, which can process SSLv3 and TLS alike.

If you can post another screenshot of the packet capture after decoding, it may help.

-Greg Wesson

Re: Inbound SSL Decryption issues

Remo — Tue, 08 Aug 2017 20:39:15 GMT

I also want to add these questions:

Does the column "decrypted" in your logs actually show "yes"?
Do you have a decryption profile applied to your decryption policy and if yes do you block sessions that cannot be decrypted and could you show a screenshot of that profile?
Did you do connection test or with ssllabs.com/htbridge.com? If yes what was the output there?

@gwesson good point with the wireshark decoding

I also looked again to the posted wireshark screenshot. And if this is the SSL session, I think there are too many packets for a failing TLS session based on a ciphersuite mismatch. But after setting the decoding session we should know more.

And last but not least: did you also capture the session aftet NAT, so between the firewall and your server? Or did you try to connect to your server directly for testing purposes? (May be with openssl directly or with a tool like sslyze)