Inbound SSL decryption - Digicert

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Reply
Highlighted
Cyber Elite

Ok, now it is getting kind of strange. Mainly because it does not matter if the F5 is configured for offloading or passthrough...

So, this is at least how I would continue here:

  • Check the details of a tls handshake from your F5 and also from a connection directly to a server
  • Compare these TLS handshakes
  • Do a packet capture on the firewall towards your F5 and the server which is not locatdd behind the firewall
  • Compare the TLS handshakes
  • If there is still no obvious reason in the handshakes you need to dig deeper: start a packet capture with also logging enabled and enable the features proxy basic and flow basic. During connection tests for connecrions towards your F5 check the global counters multiple times. Maybe there is already something. If not then aggregate the captured logs and analyze them, at latest there hopefully is the reason why the connection towards your F5 isn't working. (Probably this could be analyzed on the F5 also, but there I have no idea how it works as I never used such a loadbalancer/WAF)
Highlighted
L4 Transporter

@vsys_remo Thanks for the pointers, it is is still not working and i think i can zero in on the issue as below. First decryption process is different when when performing inbound inspection vs forward proxy. With inbound we are only eaves dropping on the session and the ciphers need to match on both sides. Thanks to this KB that explains it better. 

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClV8CAK

 

So i started looking in packet captures as you suggested, and firewall shows drop for packets with this cipher.

Also i checked from the browser with both inspection disabled/enabled and found the same ciphers. 

image.png

image.png

Although F5 or other servers that fail support other ciphers but they all seem to liken above more. Below is the cipher enum results for a server with inspection disabled, not behind F5 but it still failed. 

 

| ssl-enum-ciphers: 

|   TLSv1.0: 

|     ciphers: 

|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C

|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C

|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C

|     compressors: 

|       NULL

|     cipher preference: client

|     warnings: 

|       64-bit block cipher 3DES vulnerable to SWEET32 attack

|       Broken cipher RC4 is deprecated by RFC 7465

|       Ciphersuite uses MD5 for message integrity

|   TLSv1.1: 

|     ciphers: 

|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C

|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C

|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C

|     compressors: 

|       NULL

|     cipher preference: client

|     warnings: 

|       64-bit block cipher 3DES vulnerable to SWEET32 attack

|       Broken cipher RC4 is deprecated by RFC 7465

|       Ciphersuite uses MD5 for message integrity

|   TLSv1.2: 

|     ciphers: 

|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C

|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C

|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C

|     compressors: 

 

If i select 1.2 as the minimum protocol in decryption profile 3DES get grayed out. Which leads me to be believe "TLS_RSA_WITH_3DES_EDE_CBC_SHA" is not supported on firewall. So in all what i understand/ believe is that because ciphers need to match on both sides of firewall, which is not happening and is causing this issue. Do you think i am right at this conclusion and we need to fix on the server side ciphers that are supported. 

 

Highlighted
Cyber Elite

Hi @raji_toor 

Yes, I think you are right with this conclusion. But besides that I would recommend anyway to configure some stronger ciphersuites on the servers, you could give it a try with enabling 3DES in the decryption profile via CLI. I did not test it with an actual inbound decryption, but I was able to enable 3DES with TLS1.2 in CLI. On the WebUI then it does not even show 3DES enabled but in the configuration it is there.

Highlighted
L4 Transporter

I was able to set ciphers for a test VIP on F5, And i enabled only 1 cipher which the F5-VIP had in server hello after disabling 1.3 on it.

 

Currently this F5 VIP supports single cipher, and it works with no inspection, tested from chrome. I can enable more ciphers too but i believe this should have worked. Below is with no inspection.

image.png

image.png

When i turn inspection on i get this in chrome.

image.png

Client hello ciphers

image.png 

 

PA profile settings

 

  ssl-inbound-proxy {
    block-unsupported-version no;
    block-unsupported-cipher no;
    block-if-no-resource no;
    block-if-hsm-unavailable no;
  }
  ssl-protocol-settings {
    min-version tls1-2;
    keyxchg-algo-rsa yes;
    keyxchg-algo-dhe yes;
    enc-algo-3des no;
    enc-algo-rc4 no;
    auth-algo-sha1 yes;
    keyxchg-algo-ecdhe yes;
    enc-algo-aes-128-cbc yes;
    enc-algo-aes-256-cbc yes;
    enc-algo-aes-128-gcm yes;
    enc-algo-aes-256-gcm yes;
    auth-algo-sha256 yes;
    auth-algo-sha384 yes;

 

 

What do you suggest @vsys_remo 

 

Highlighted
L4 Transporter

I further enabled more ciphers on VIP and i still get cipher mismatch, while turning inspection off it works

 

With no inspection

TLSv1.2: 
| ciphers: 
| TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (dh 1024) - A 
| TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (dh 1024) - A 
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A 
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A

 

 

Highlighted
Cyber Elite

Hi @raji_toor 

You now reached a point where it is at least possible, that something on the firewall ist not compatible with the F5. So at this point I would recommend to open a support case and then continue with the following troubleshooting (these logs will also be required in the support case). 

Obbiously you need to change the IPs and maybe also the port, depending on your configuration

  • clear counter global
  • debug dataplane packet-diag clear all
  • debug dataplane packet-diag clear log log
  • debug dataplane packet-diag set filter match source 1.1.1.1 destination 2.2.2.2 destination-port 443 protocol 6
  • debug dataplane packet-diag set log feature proxy basic
  • debug dataplane packet-diag set log feature flow basic
  • debug dataplane packet-diag set log on
  • debug dataplane packet-diag set capture on

Then you connect to the VIP with decryption enabled and right after that enter the following command. In the output, maybe you already see a specific counter which could lead to the reason of the problem

  • show counter global filter packet-filter yes

Try to connect a second time and then stop the logging and capture

  • debug dataplane packet-diag set log off
  • debug dataplane packet-diag set capture off

Then aggregate the logs. The output of the command will show you the filename that you need to analyze

  • debug dataplane packet-diag aggregate-logs

Prior to analyze the logfile start now with generating a techsupportfile (for the supportcase)

Maybe for analysis you want to copy the logile away from the firewall to open it in a texteditor but of course you can also view it in cli. About here I don't know what to do exactly, I would scroll through the logs to find something that maybe shows the reason why the TLS handshake fails after the client hello.

 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!