topic Re: Sectigo CA Chain Decryption Issues in General Topics

Sectigo CA Chain Decryption Issues

P.Carroll — Mon, 01 Jun 2020 10:40:58 GMT

Due to the recent expiration of the Sectigo RSA CA cert (https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020) and our Palo firewall SSL decryption policy configuration to block expired certificates we are noticing that any website that is publishing the old expired CA chain (for example netaoc.org.uk) is being blocked due to them publishing an expired cert.

This is obviously working as expected however it's difficult for me to come into contact with each website hosting one of these invalid CA chains to get them to resolve the issue while our users experience issues and I manually exclude the sites from decryption. I of course could turn off expired certificate blocking however this something I would rather not do.

I have noticed that web browsers like Chrome when not running through decryption are handling this issue just fine as they seem to look up the new correct CA certificate themselves and use that. Is there a way I can configure out Palo to act in the same way or am I stuck being reliant on the web admins of the individual sites to correct their chain issues?

Re: Sectigo CA Chain Decryption Issues

RobinClayton — Mon, 01 Jun 2020 11:46:55 GMT

Yes we are seeing this issue

Some customers are reporting it too when accessing one of our websites, but that's an external problem

I can't replicate it accessing our site externally.

I don't think it's necessarily a web hoster problem, our chain looks valid, and the certificate was only generated with it's chain in December.

I have logged a support case, I suggest you do the same.

Re: Sectigo CA Chain Decryption Issues

P.Carroll — Mon, 01 Jun 2020 12:08:59 GMT

The only reason I think it's a chain issue from the sites host is if you check the website with a tool like https://whatsmychaincert.com/ it will report that the site is delivering an invalid chain but it implies that modern web browsers will transparently fix this issue for the end user.

Re: Sectigo CA Chain Decryption Issues

RobinClayton — Mon, 01 Jun 2020 13:29:05 GMT

The fact that https://support.sectigo.com fails as well leads me to believe that the test site is not able to correctly process the request.

Re: Sectigo CA Chain Decryption Issues

kaschekotov — Mon, 01 Jun 2020 14:07:52 GMT

Also, please note: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g0000008UFBCA2

Re: Sectigo CA Chain Decryption Issues

Sec101 — Mon, 01 Jun 2020 14:40:06 GMT

This is an issue for us too. Why isn't Palo updating these? Does the palo check all the chains for these, apparently Sectigo is saying that the cross signed certificate is enough to stop an error for these?

Re: Sectigo CA Chain Decryption Issues

RobinClayton — Mon, 01 Jun 2020 15:17:42 GMT

From PA Support

"From a quick search, I was able to see that multiple issues have been reported with respect to Sectigo Certificates Expiration.

As a workaround,you can either allow untrust cert or exclude the website from decryption which is causing issue.

PA has updated with the latest CA certification, so as of now no action needed on PA certificate store. If server chain is not updated till now then that might cause issue here.

Please let me know if you have any further queries or concerns regarding the case."

Awaiting their guidance on why the store is not up to date.

Rob

Re: Sectigo CA Chain Decryption Issues

kalakai — Mon, 01 Jun 2020 21:30:37 GMT

According to this twitter thread, the issue is with how OpenSSL handles (or doesn't handle) validating certificate chains. I believe that the PAN firewalls use OpenSSL for certificate validations which is why the firewall fails to see that the server's certificate is actually valid.

As a workaround, this is what we did:

Create a custom URL category "Expired Certificate Bypass"
Objects > Custom Objects > URL Category
Clone the decryption profile object and uncheck "Block sessions with expired certificates"
Objects > Decryption > Decryption Profile
Clone the decryption rule and use the new decryption profile and url category you just created
Policies > Decryption

This is not ideal or scalable but it works for business critical sites.

Re: Sectigo CA Chain Decryption Issues

benlangberg — Tue, 02 Jun 2020 04:32:05 GMT

We had been running with a separate exclusion list, our helpdesk became overwhelmed with requests and cert issues. As of this post we have disabled "Block sessions with expired certificates"

Monitoring this discussion for further updates.

Re: Sectigo CA Chain Decryption Issues

thomaswiesner — Tue, 02 Jun 2020 06:39:47 GMT

I will do exactly the same "Benlangberg"

Re: Sectigo CA Chain Decryption Issues

OwenFuller — Tue, 02 Jun 2020 15:13:57 GMT

As @kalakai said, this problem appears to be related to OpenSSL incorrectly building the trust chain for certificates which used to chain up to the now expired Sectigo CA cert. This article (which we're also providing to TAC on our own case) provides a very detailed explanation for those who are interested: https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-external-root-expiration-may-2020

Re: Sectigo CA Chain Decryption Issues

OwenFuller — Tue, 02 Jun 2020 15:36:43 GMT

According to the KB article TAC referenced (https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g0000008UFBCA2), the new CA certs are already in the CA store on the firewalls. It's saying that the problem is due to "Some servers that are using certificates signed by these CAs are still including the expired CAs as part of certification chain supplied to the client." It also mentions that "Our NGFW Trusted CA store is already updated with the self-signed certs, and no change is needed to Trusted CA store on PA." However, given that the chains build properly on other systems, and there are know issues with certain clients (see my previous comment), this still seems to be a problem with PANOS (a la OpenSSL) in my opinion. We're going to continue working w/ TAC for answers.

Re: Sectigo CA Chain Decryption Issues

RobinClayton — Tue, 02 Jun 2020 15:44:39 GMT

I have expiration turned off at present , support are not being very clear. will turn it on again later and see if the cert store has updated.

Re: Sectigo CA Chain Decryption Issues

Tarcizoa — Tue, 02 Jun 2020 16:24:05 GMT

From what i see the servers have both chains. PaloAlto behavior is one of the following:

1 - It checks if any expired on the server and block no matter if one is good.

2 - It only check the first one(expired) and doesn't even check the second one.

I agree with you that it should be fixed but looks like its more a code change then a certificate chain issue.

The article that you posted previously shows that clearly on option 2.

Someone was able to reboot firewall just to validate if its not a cache or something like?

Re: Sectigo CA Chain Decryption Issues

OwenFuller — Tue, 02 Jun 2020 17:00:32 GMT

We only saw about ~50 destinations w/ decrypt-cert-validation as the session-end-reason in our logs, and several seem to be junk we don't really care about. We're opting to leave expiration on for most user traffic. We've created a custom URL category for affected sites which are business-related, and are using it in a separate SSL decryption policy that doesn't block the expired cert. Seems to be the best option to avoid completely disabling cert expiration checking for our purposes.

Re: Sectigo CA Chain Decryption Issues

Tarcizoa — Tue, 02 Jun 2020 17:04:16 GMT

We proceeded almost the same, only difference is we're using minemeld with a dynamic list so we dont need to push on the firewall at each addition on the list.

Re: Sectigo CA Chain Decryption Issues

kbe — Tue, 02 Jun 2020 17:22:18 GMT

Same problem here.

Right now i only saw 1 site not working, but i guess more will follow.

I also set up a new decrypt policy with a new decrypt profile allowing expired certs and put custom url list in place as a workaround.

Re: Sectigo CA Chain Decryption Issues

OwenFuller — Tue, 02 Jun 2020 18:32:00 GMT

Looks like the official advisory is out, and the suggestions are to do the exemptions like most of us have been talking about already: https://live.paloaltonetworks.com/t5/customer-advisories/decryption-errors-created-by-the-expired-addtrust-external-root/ta-p/330976 What a mess. Good luck w/ your exemptions everyone!

Re: Sectigo CA Chain Decryption Issues

kalakai — Tue, 02 Jun 2020 18:46:27 GMT

I like @Tarcizoa's idea of using an external dynamic list to manage these exceptions. We don't have minemeld but we can host a text file on an internal web server and just update the text file with any new exceptions. Then set the EDL to check for updates every 5 minutes. Sounds better than having to do commits every time a new site is discovered.

Re: Sectigo CA Chain Decryption Issues

axemte — Wed, 03 Jun 2020 11:50:46 GMT

I received an update from TAC saying they also have an engineering request for this issue to identify if the PA behavior can be changed to accept the best root CA instead of the ones which are expired. They will keep us posted with the coming updates.