As @kalakai said, this problem appears to be related to OpenSSL incorrectly building the trust chain for certificates which used to chain up to the now expired Sectigo CA cert. This article (which we're also providing to TAC on our own case) provides a very detailed explanation for those who are interested: https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-ex...
According to the KB article TAC referenced (https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g0000008UFBCA2), the new CA certs are already in the CA store on the firewalls. It's saying that the problem is due to "Some servers that are using certificates signed by these CAs are still including the expired CAs as part of certification chain supplied to the client." It also mentions that "Our NGFW Trusted CA store is already updated with the self-signed certs, and no change is needed to Trusted CA store on PA." However, given that the chains build properly on other systems, and there are know issues with certain clients (see my previous comment), this still seems to be a problem with PANOS (a la OpenSSL) in my opinion. We're going to continue working w/ TAC for answers.
I have expiration turned off at present , support are not being very clear. will turn it on again later and see if the cert store has updated.
From what i see the servers have both chains. PaloAlto behavior is one of the following:
1 - It checks if any expired on the server and block no matter if one is good.
2 - It only check the first one(expired) and doesn't even check the second one.
I agree with you that it should be fixed but looks like its more a code change then a certificate chain issue.
The article that you posted previously shows that clearly on option 2.
Someone was able to reboot firewall just to validate if its not a cache or something like?
We only saw about ~50 destinations w/ decrypt-cert-validation as the session-end-reason in our logs, and several seem to be junk we don't really care about. We're opting to leave expiration on for most user traffic. We've created a custom URL category for affected sites which are business-related, and are using it in a separate SSL decryption policy that doesn't block the expired cert. Seems to be the best option to avoid completely disabling cert expiration checking for our purposes.
We proceeded almost the same, only difference is we're using minemeld with a dynamic list so we dont need to push on the firewall at each addition on the list.
Same problem here.
Right now i only saw 1 site not working, but i guess more will follow.
I also set up a new decrypt policy with a new decrypt profile allowing expired certs and put custom url list in place as a workaround.
Looks like the official advisory is out, and the suggestions are to do the exemptions like most of us have been talking about already: https://live.paloaltonetworks.com/t5/customer-advisories/decryption-errors-created-by-the-expired-ad... What a mess. Good luck w/ your exemptions everyone!
I like @Tarcizoa's idea of using an external dynamic list to manage these exceptions. We don't have minemeld but we can host a text file on an internal web server and just update the text file with any new exceptions. Then set the EDL to check for updates every 5 minutes. Sounds better than having to do commits every time a new site is discovered.
I received an update from TAC saying they also have an engineering request for this issue to identify if the PA behavior can be changed to accept the best root CA instead of the ones which are expired. They will keep us posted with the coming updates.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!
The Live Community thanks you for your participation!