SSL inspection issues with PAN-OS 10.2.3

[itassetbenilde]

Good day,

Hoping to get some insights on a particular issue we're having.

I've managed to get SSL inspection running using a test server:

- uploaded the private key and certificate, and the CA's public certificate

- created a decryption profile and decryption policy 

While it tested OK, i can't seem to get it running on our production servers. The symptoms are:

- on the client side, the server seems to just hang when GET'ing the / page

- no decryption errors registered

- upon checking the packet capture, it seems that the TLS communication stops after the Client Hello phase - no Server Hello packet received. 

- already tried using the private key and certificate from the production server to the test server, and it worked just fine.

I am fairly sure about my security policies -- the production server splash page loads up immediately after disabling the decryption policy.

Any suggestions on what the problem might be? To mention, the production server is running IIS. tia

[OtakarKlier]

Hello,

The server has to trust the certificate that you are using for ssl decryption. If this is an AD certificate, you'll need a subordinate certificate, or install the certificate you are using for ssl decryption on each server.

Regards,

[BPry]

@itassetbenilde,

Can you clarify if you're talking about SSL forward proxy (traditional decryption) or SSL inbound inspection? You're initial post makes me thing that you're talking about inbound inspection, and the issue that you're describing would have me looking at cipher suites and algorithms and make sure that you don't have a mismatch in what the firewall supports and what's being offered by the server. 

[itassetbenilde]

Thanks for the replies

To clarify:

1. i don't think it's a certificate issue -- i used the same set of key/certificate on my test server and it worked just fine.

2. Yes, i am trying to get SSL inbound inspection to work.

Some updates:

1. The problem seems to be tied to the DHE/ECDHE key exchange algorithm in the decryption profile. When i set it to just RSA it works just fine.

2. On wireshark it's clear that client and server are able to establish a TLS connection, and i can see the ECDHE settings. i can also see the server sending "application data" back to the client, then it just hangs until the client sends a reset.

3. Curious thing is that the page doesn't load when i use browsers, but i can access the page using curl and/or python requests. The only difference i could spot is that traffic is detected(on wireshark) as "Hypertext Transfer Protocol 2" when i used the browser, and "Hypertext Transfer Protocol" when  i used curl.

[itassetbenilde]

Am starting to suspect this is an IIS configuration thing. i've managed to get it working on a second server running Apache.

i've already tried to set the SSL cipher suite via group policy settings to match the SSL decryption policy, still no go.

Any suggestions on what to check next?

[itassetbenilde]

I dug a little deeper into the matter. While investigating a different issue, i came across a suggestion to enable the "Strip ALPN" feature in the SSL Forward Proxy settings. SSL inbound inspection worked with DHE and ECDHE key algos enabled.

This seems to be an issue with HTTP/2. When HTTP/2 is enabled on both IIS and Apache, the connection hangs unless i enable "Strip ALPN" and force the connection down to HTTP/1.1.

On the upside, glad to have DHE/ECDHE key algos enabled...but are there any risks to forcing HTTP/1.1 ? Thanks

[itassetbenilde]

Just to give this issue some closure.

It turns out there is a known bug with PANOS 10.2.3 regarding HTTP/2 streams(PAN-PAN-206005), and we've been advised to upgrade.