Sectigo CA Chain Decryption Issues

[P.Carroll]

Due to the recent expiration of the Sectigo RSA CA cert (https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-202...) and our Palo firewall SSL decryption policy configuration to block expired certificates we are noticing that any website that is publishing the old expired CA chain (for example netaoc.org.uk) is being blocked due to them publishing an expired cert.

This is obviously working as expected however it's difficult for me to come into contact with each website hosting one of these invalid CA chains to get them to resolve the issue while our users experience issues and I manually exclude the sites from decryption.  I of course could turn off expired certificate blocking however this something I would rather not do.

I have noticed that web browsers like Chrome when not running through decryption are handling this issue just fine as they seem to look up the new correct CA certificate themselves and use that.  Is there a way I can configure out Palo to act in the same way or am I stuck being reliant on the web admins of the individual sites to correct their chain issues?

[RobinClayton]

Yes we are seeing this issue

Some customers are reporting it too when accessing one of our websites, but that's an external problem

I can't replicate it accessing our site externally.

I don't think it's necessarily a web hoster problem, our chain looks valid, and the certificate was only generated with it's chain in December.

I have logged a support case, I suggest you do the same.

[P.Carroll]

The only reason I think it's a chain issue from the sites host is if you check the website with a tool like https://whatsmychaincert.com/ it will report that the site is delivering an invalid chain but it implies that modern web browsers will transparently fix this issue for the end user.

[RobinClayton]

The fact that https://support.sectigo.com fails as well leads me to believe that the test site is not able to correctly process the request.

[kaschekotov]

Also, please note: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g0000008UFBCA2

[Sec101]

This is an issue for us too.   Why isn't Palo updating these?   Does the palo check all the chains for these, apparently Sectigo is saying that the cross signed certificate is enough to stop an error for these?

[RobinClayton]

From PA Support

"From a quick search, I was able to see that multiple issues have been reported with respect to Sectigo Certificates Expiration.

As a workaround,you can either allow untrust cert or exclude the website from decryption which is causing issue.

PA has updated with the latest CA certification, so as of now no action needed on PA certificate store. If server chain is not updated till now then that might cause issue here.

Please let me know if you have any further queries or concerns regarding the case."

Awaiting their guidance on why the store is not up to date.

Rob

[kalakai]

According to this twitter thread, the issue is with how OpenSSL handles (or doesn't handle) validating certificate chains. I believe that the PAN firewalls use OpenSSL for certificate validations which is why the firewall fails to see that the server's certificate is actually valid.

As a workaround, this is what we did:

1. Create a custom URL category "Expired Certificate Bypass"
   Objects > Custom Objects > URL Category
2. Clone the decryption profile object and uncheck "Block sessions with expired certificates"
   Objects > Decryption > Decryption Profile
3. Clone the decryption rule and use the new decryption profile and url category you just created
   Policies > Decryption

This is not ideal or scalable but it works for business critical sites.

[benlangberg]

We had been running with a separate exclusion list, our helpdesk became overwhelmed with requests and cert issues. As of this post we have disabled "Block sessions with expired certificates"

Monitoring this discussion for further updates. 

[thomaswiesner]

I will do exactly the same "Benlangberg"

[OwenFuller]

As @kalakai said, this problem appears to be related to OpenSSL incorrectly building the trust chain for certificates which used to chain up to the now expired Sectigo CA cert.  This article (which we're also providing to TAC on our own case) provides a very detailed explanation for those who are interested:  https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-ex...

[OwenFuller]

According to the KB article TAC referenced (https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g0000008UFBCA2), the new CA certs are already in the CA store on the firewalls.  It's saying that the problem is due to "Some servers that are using certificates signed by these CAs are still including the expired CAs as part of certification chain supplied to the client." It also mentions that "Our NGFW Trusted CA store is already updated with the self-signed certs, and no change is needed to Trusted CA store on PA."  However, given that the chains build properly on other systems, and there are know issues with certain clients (see my previous comment), this still seems to be a problem with PANOS (a la OpenSSL) in my opinion.  We're going to continue working w/ TAC for answers.

[RobinClayton]

I have expiration turned off at present , support are not being very clear. will turn it on again later and see if the cert store has updated.

[Tarcizoa]

From what i see the servers have both chains. PaloAlto behavior is one of the following:

1 - It checks if any expired on the server and block no matter if one is good.

2 - It only check the first one(expired) and doesn't even check the second one.

I agree with you that it should be fixed but looks like its more a code change then a certificate chain issue.

The article that you posted previously shows that clearly on option 2.

Someone was able to reboot firewall just to validate if its not a cache or something like?

[OwenFuller]

We only saw about ~50 destinations w/ decrypt-cert-validation as the session-end-reason in our logs, and several seem to be junk we don't really care about.  We're opting to leave expiration on for most user traffic.  We've created a custom URL category for affected sites which are business-related, and are using it in a separate SSL decryption policy that doesn't block the expired cert.  Seems to be the best option to avoid completely disabling cert expiration checking for our purposes.