SSL decryption - How to deal with third party sites that don't install the intermediate certificate?

[scottsander]

I turned on TLS (let's start calling it what it is) decryption for our IT personnel only a couple of years ago.  It was considered a pilot and I always planned to work with our legal department to craft a policy and start rolling it out to the broader organization; however, it seems like every time I get ready to do that, one of our IT users reports a website that is blocked by the PAN NGFW (running PAN-OS 6.1.10) due to a certificate issue where the status is "untrusted".

In 100% of these cases, the certificate is untrusted because the web server hosting the site in question doesn't have the intermediate certificate installed and it is impossible to reach the site unless I exclude if from decryption.

Sure, I could just mark the site as excluded from decryption but I tend to try to get the external organization to fix the problem.  This has been a huge burden.  Despite the fact that I bundle significant evidence of the issue from publicly available TLS testing tools that clearly explain the issue, most external organizations are initially incredulous that they have a configuration problem.  It's the same thing every time: they say we are the only ones reporting any sort of access issue and that it must be something on our side.  It can be several weeks of back and forth before they finally understand the issue and fix it.  In some cases, they simply refuse to fix it or I can't successfully navigate their support organization to get to someone who understands the issue and has the power to do anything about it.

I can't imagine what it would be like if I ended up rolling TLS decryption out to the entire organization.  How many of these issues would our users encounter on a daily basis?  How many painful discussions would I need to have with first level support personnel telling me reboot my computer and clear cookies followed painful discussions with IT managers or systems people that say no one else has the problem but us?

What is your experience in rolling out TLS decryption to a wide user base and dealing with third party web servers missing the intermediate certificate?  Do I have something configured incorrectly that is causing me all of this grief?

[Raido_Rattameister]

Can you give some example websites that don't work for you?

One thing you can try is to install this root and intermediate into Palo. This should fix the issue aswell.

[scottsander]

At this time, I'm not aware of any sites where we are having this problem.

I am aware that I could install the sites' root and intermediate certificates and this will "solve" the problem, at least on our end.  But the true problem is on the web sites' end and my view is that they should fix their misconfigured site.

I have the "Block sessions with untrusted issuers" box checked in our decryption profile.  I think that's what results in this issue; however, I'm hesitant unchecking that box, which I would interpret as meaning that we allow sessions with untrusted issuers.  I can't see how that would be desirable.

[nikoo]

I have not used that in large deployments with strict enterprise security policies, and I hope I got your issue correctly, but what is your general security policy if you are not using PA for decryption? How it goes in a classic way - users get "not trusted" warning for sites without properly signed cert and, if user is willing to continue, certificate will still have to be accepted. Generrally in case Palo sits in the middle, you can probably create two certs - one for signing trusted entities (Forward trust), so users will not get a warning, and second one for untrusted sites (Forward untrust), so the user can still get the warning and choose to accept/reject that connection further. 

And having to impact third party is way too much to ask in general - you should not be responsible for their mess unless there is a really good reason for that. They are not paying you salary for that. 🙂

[scottsander]

I'll have to look into the forward untrust thing.  I was not aware of that.  Perhaps it is a solution.  Today, for connections which are not decrypted, people do receive the normal browser certificate warning and are permitted to bypass it.

That said, I do not like to train users to bypass security warnings.  I think that sets a bad precedent.  You might say that we are already doing this today, but that's not really true.  Browsers themselves handle sites better when the intermediate certificate is missing.  PAN-OS just chokes in these situations..

[nikoo]

In general - yes, reasonable sites should have proper and trusted certificate warning. You can probably play around with decryption policies and create a list of sites (with URL category) which should be accessed by users, but uses untrusted cert and, when they are passing via Palo, resign them with your trusted CA. That way users will still be able to access these sites, they will not have cert warning and you will control the list of such a sites.

Browser vs Palo thing - I guess it imay be a matter of trusted CA list. Device -> Certificates -> Default Trusted Certificate Authorities. They probably differ, so that may impact the result.

[scottsander]

No, that's incorrect.  PAN-OS trusts the root certificates but, unlike how browsers handle the situation, it doesn't trust the host certificate unless the host also has the intermediate certificate installed.

For such sites, users not being decrypted do not experience any certificate warning at all because their browsers process the certificate chain differently (I believe they use a different mechanism to acquire the missing intermediate certificate).

The challenge is how PAN-OS handles the missing intermediate certificate vs. how browsers handle it.

[mb_equate]

6 years on and this is still an issue - while the server owner should be responsible for serving the intermediate cert, most modern browsers find missing certs from the AIA extension (https://www.rfc-editor.org/rfc/rfc5280#section-4.2.2.1) making broken chains transparent to end users.

Example: https://support.poly.com 

In 2022, a PAN-OS 10.1 device with TLS decryption will still not trust a broken chain making the user experience resulting in sites that were previously verified and trusted being inaccessible to users without either creating exceptions or installing intermediate certs in PAN-OS. Or forcing the server owner to fix it. Neither of which anyone wants to manage.

PAN-OS needs to provide the option to fetch intermediates in the same manner it does CRLs (from certificate extensions) and use the CRL service route.

[NEAIT]

I've just suickly read the 11.0 releases notes, and I haven't found anything about taking care of this problem 😞
Has anyone been in touch with PA support ? Any official answer on what to expect ?

It is always difficult to explain to our users why they can't access a site from the company network while they can from their mobile phone or from home.