topic Re: Inbound SSL decryption - Digicert in General Topics

Inbound SSL decryption - Digicert

raji_toor — Tue, 11 Aug 2020 21:47:36 GMT

If inbound SSL inspection when using Digicert certificate is not supported, what is the alternative. We have many web-servers using same wildcard cert used for GlobalProtect and wanted use this same certificate but it doesn't work. Is there any other mechanism to implement inbound SSL inspection.

Re: Inbound SSL decryption - Digicert

Remo — Tue, 11 Aug 2020 21:50:10 GMT

Hi @raji_toor

What do you mean with not supported with digicert? What type (algorythms) of certificate do you have?

Re: Inbound SSL decryption - Digicert

raji_toor — Tue, 11 Aug 2020 23:24:49 GMT

@Remo I think this is not the only KB i came across while to work this out https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClEZCA0

NOTE: Because SSL certificate providers such as Entrust, Verisign, Digicert, and GoDaddy do not sell CAs, they are not supported in SSL Decryption.

Re: Inbound SSL decryption - Digicert

raji_toor — Tue, 11 Aug 2020 23:27:53 GMT

Re: Inbound SSL decryption - Digicert

Remo — Wed, 12 Aug 2020 06:40:08 GMT

Hi @raji_toor

For SSL inbound decryption the servercertificates including your wildcardcertificate is supported. Only for SSL forward proxy you have to use a self signed certificate.

Re: Inbound SSL decryption - Digicert

raji_toor — Wed, 12 Aug 2020 15:02:03 GMT

@Remo Its not working for me. certificate in use by GP

certificate on web-server

PA config

With above profile attached to the decryption rule

With no profile attached to the decryption rule

traffic from outside

Re: Inbound SSL decryption - Digicert

jdelio — Wed, 12 Aug 2020 16:44:38 GMT

Quote:

NOTE: Because SSL certificate providers such as Entrust, Verisign, Digicert, and GoDaddy do not sell CAs, they are not supported in SSL Decryption.

I think the way that it was intended was that SSL Forward Proxy (standard decryption) is not supported with SSL Certificate Providers because they DO NOT SELL A CA (Certificate Authority), in other words, they do not sell you a license to create certificates that look like they created the certificate, You would then have the ability to create SSL Certs from all of these providers (and maybe try to sell them).

SSL Inbound Decryption, where you are intercepting traffic to an internal server and therefore use that SSL Cert to be installed on the Firewall to "Impersonate" the internal server.. that can be a Certificate from any provider.. because in that scenario, no SSL Certs are being created.

I hope that makes a little more sense.

Re: Inbound SSL decryption - Digicert

raji_toor — Wed, 12 Aug 2020 17:56:19 GMT

@jdelio Thanks for explanation. As i replied above to @Remo with my settings and it is failing. Not sure what else i need to fix.

I have also imported the cert given by our web-admin and using that still fails. If I unselect below options then no decryption happens but website opens.

Re: Inbound SSL decryption - Digicert

Remo — Wed, 12 Aug 2020 19:54:39 GMT

Hi @raji_toor

Now you need to start troubleshooting the connection. As you are using the same certificate already for globalprotect, this is not the reason for this issue. First check what ciphersuites and tls versions does the server offer for a tls connection. With a website like Qualys ssllabs - as you already know - you can check what the server does offer (for this test of course you need to disble decryption). Is the server behind the firewall a windows server? If yes, which version and do you have extended master secret enabled (is enabled by default)?

Re: Inbound SSL decryption - Digicert

raji_toor — Wed, 12 Aug 2020 22:25:54 GMT

@Remo server supports a lot, and almost all are linux servers

TLSv1.0 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLSv1.1 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLSv1.2 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA TLS_DHE_RSA_WITH_AES_256_CBC_SHA TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA TLS_DHE_RSA_WITH_AES_128_CBC_SHA TLS_RSA_WITH_AES_256_GCM_SHA384 TLS_RSA_WITH_AES_128_GCM_SHA256 TLS_RSA_WITH_AES_128_CBC_SHA256 TLS_RSA_WITH_AES_256_CBC_SHA TLS_RSA_WITH_AES_128_CBC_SHA TLSv1.3 TLS_AES_256_GCM_SHA384 TLS_CHACHA20_POLY1305_SHA256 TLS_AES_128_GCM_SHA256

Decryption profile allows for below

Atleast 3 of them are common between the profile and what server supports under TLS1.2

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

-----------------------------------

I also realized that inbound inspection broke for all the servers sitting behind F5 in DMZ. Even though F5 VIP is configured either for SSL offload or SSL pasthrough.

So i found a server which was not behind F5 and decryption worked.

Re: Inbound SSL decryption - Digicert

Remo — Wed, 12 Aug 2020 23:35:14 GMT

Ok, now it is getting kind of strange. Mainly because it does not matter if the F5 is configured for offloading or passthrough...

So, this is at least how I would continue here:

Check the details of a tls handshake from your F5 and also from a connection directly to a server
Compare these TLS handshakes
Do a packet capture on the firewall towards your F5 and the server which is not locatdd behind the firewall
Compare the TLS handshakes
If there is still no obvious reason in the handshakes you need to dig deeper: start a packet capture with also logging enabled and enable the features proxy basic and flow basic. During connection tests for connecrions towards your F5 check the global counters multiple times. Maybe there is already something. If not then aggregate the captured logs and analyze them, at latest there hopefully is the reason why the connection towards your F5 isn't working. (Probably this could be analyzed on the F5 also, but there I have no idea how it works as I never used such a loadbalancer/WAF)

Re: Inbound SSL decryption - Digicert

raji_toor — Thu, 13 Aug 2020 23:17:33 GMT

@Remo Thanks for the pointers, it is is still not working and i think i can zero in on the issue as below. First decryption process is different when when performing inbound inspection vs forward proxy. With inbound we are only eaves dropping on the session and the ciphers need to match on both sides. Thanks to this KB that explains it better.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClV8CAK

So i started looking in packet captures as you suggested, and firewall shows drop for packets with this cipher.

Also i checked from the browser with both inspection disabled/enabled and found the same ciphers.

Although F5 or other servers that fail support other ciphers but they all seem to liken above more. Below is the cipher enum results for a server with inspection disabled, not behind F5 but it still failed.

If i select 1.2 as the minimum protocol in decryption profile 3DES get grayed out. Which leads me to be believe "TLS_RSA_WITH_3DES_EDE_CBC_SHA" is not supported on firewall. So in all what i understand/ believe is that because ciphers need to match on both sides of firewall, which is not happening and is causing this issue. Do you think i am right at this conclusion and we need to fix on the server side ciphers that are supported.

Re: Inbound SSL decryption - Digicert

Remo — Fri, 14 Aug 2020 08:16:04 GMT

Hi @raji_toor

Yes, I think you are right with this conclusion. But besides that I would recommend anyway to configure some stronger ciphersuites on the servers, you could give it a try with enabling 3DES in the decryption profile via CLI. I did not test it with an actual inbound decryption, but I was able to enable 3DES with TLS1.2 in CLI. On the WebUI then it does not even show 3DES enabled but in the configuration it is there.

Re: Inbound SSL decryption - Digicert

raji_toor — Fri, 14 Aug 2020 17:34:06 GMT

I was able to set ciphers for a test VIP on F5, And i enabled only 1 cipher which the F5-VIP had in server hello after disabling 1.3 on it.

Currently this F5 VIP supports single cipher, and it works with no inspection, tested from chrome. I can enable more ciphers too but i believe this should have worked. Below is with no inspection.

When i turn inspection on i get this in chrome.

Client hello ciphers

PA profile settings

ssl-inbound-proxy { block-unsupported-version no; block-unsupported-cipher no; block-if-no-resource no; block-if-hsm-unavailable no; } ssl-protocol-settings { min-version tls1-2; keyxchg-algo-rsa yes; keyxchg-algo-dhe yes; enc-algo-3des no; enc-algo-rc4 no; auth-algo-sha1 yes; keyxchg-algo-ecdhe yes; enc-algo-aes-128-cbc yes; enc-algo-aes-256-cbc yes; enc-algo-aes-128-gcm yes; enc-algo-aes-256-gcm yes; auth-algo-sha256 yes; auth-algo-sha384 yes;

What do you suggest @Remo

Re: Inbound SSL decryption - Digicert

raji_toor — Fri, 14 Aug 2020 18:40:44 GMT

I further enabled more ciphers on VIP and i still get cipher mismatch, while turning inspection off it works

With no inspection

Re: Inbound SSL decryption - Digicert

Remo — Sat, 15 Aug 2020 11:55:12 GMT

Hi @raji_toor

You now reached a point where it is at least possible, that something on the firewall ist not compatible with the F5. So at this point I would recommend to open a support case and then continue with the following troubleshooting (these logs will also be required in the support case).

Obbiously you need to change the IPs and maybe also the port, depending on your configuration

clear counter global
debug dataplane packet-diag clear all
debug dataplane packet-diag clear log log
debug dataplane packet-diag set filter match source 1.1.1.1 destination 2.2.2.2 destination-port 443 protocol 6
debug dataplane packet-diag set log feature proxy basic
debug dataplane packet-diag set log feature flow basic
debug dataplane packet-diag set log on
debug dataplane packet-diag set capture on

Then you connect to the VIP with decryption enabled and right after that enter the following command. In the output, maybe you already see a specific counter which could lead to the reason of the problem

show counter global filter packet-filter yes

Try to connect a second time and then stop the logging and capture

debug dataplane packet-diag set log off
debug dataplane packet-diag set capture off

Then aggregate the logs. The output of the command will show you the filename that you need to analyze

debug dataplane packet-diag aggregate-logs

Prior to analyze the logfile start now with generating a techsupportfile (for the supportcase)

Maybe for analysis you want to copy the logile away from the firewall to open it in a texteditor but of course you can also view it in cli. About here I don't know what to do exactly, I would scroll through the logs to find something that maybe shows the reason why the TLS handshake fails after the client hello.