PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

[Gertjan-HFG]

Yesterday i upgraded my pa vm-100 from panos-7.01 to 7.02.

After that facebook stopped working with SSL decryption on.

After some testing and troubleshooting this seems to be the problem.

The problem is that some akamai domains that facebook uses gives me an palo alto certificate untrusted page.

for example this domain: https://fbcdn-profile-a.akamaihd.net

The strange thing is all the certificates used by this domain are already in de PA trusted cert auth list.

Just to be sure i downloaded the certs and added them manually to the PA, but no difference.

After spending 2 hours debugging en trying to get it work,

off course i can exclude those domains from decryption or or let the PA ingnore untrusted certs but thats not the way to do it. i downgraded to panos 7.0.1 and the untrusted cert problem dissapeared.

Are more people having this issue? i think there are more sites that stop working after the upgrade.

Does anyone found a solution?

[Lucky]

Hi Gertjan,

Welcome to the community. That is an interesting display of the problem you have.

First things first, link you shared is https but apparently is not encrypted, there is no certificate attached to it? At least from my browser, I am lazy to check with curl. Do you have any other sample URLs that didn't work?

Secondly, if you found out there is a certificate - did you check on the issuer? It should be in the list of trusted certificates. I am thinking, if you can find a root certificate that signed those untrusted certificates, and install it to your device, afterwards it's signed certs will be trusted. Of course, this is only in the case you really are sure of aforementioned root certificate validity...

Did you try any of that? I haven't moved to 7.0.2 yet and I am not decrypting ATM but I would test it, provided you have valid URL 🙂

Regards

Luciano

[Gertjan-HFG]

Here is a "problem" link from the palo alto facebook page:

https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfp1/v/t1.0-1/p160x160/11196343_862388790520989_315...

  

in panos 7.0.1 its works with SSL decryption after  upgrade to 7.0.2 you get a certificate untrusted page from the Palo Alto,

Downgrade to 7.0.1 link works correct again.

I kown this page very well because we get montly request from our users who get this en then we add the root certificate on the Palo Alto and after that its working fine.

All the certificates used were already  in the default PA trusted root, just to be sure I downloaded these certificates en installed them on the device but no difference. This is for 99% sure a panos-7.0.2 bug

[pulukas]

Thanks for the pretty complete testing.  I agree this is likely a bug.

Did you open an official support ticket so this can be logged in the bug database and fixed?

These forums are informal community support.  You do need an official ticket to get the bug report created.

[jochristian]

Hi,

I'm seeing the same problem.
I have already opened an case with TAC on this.

/Jo Christian

[Gertjan-HFG]

Ok, I did not open a case with TAC yet.

Did you have problems on other sites than facebook? 

Please keep us posted!

[Lucky]

Hi Gertjan,

if you have support account, it is better if you open the case as well and point out to this discussion to the TAC engineer that takes the case; more cases = more reports on the bug = bigger weigth on fixing it 🙂 They will note it and add it to the already opened bug report.

Regards

Luciano

[jochristian]

Hi,

So far we have only seen this on Facebook. 
But we did not test many other webpages before we downgraded to 7.0.1 (that works fine).

If you open an support case on this please refer to casenumber: 00371068 

/Jo Christian

[ITCMPHC]

Interesting as I too upgraded to 7.0.2 on my 3050s last night and I'm not seeing this issue on the page you linked. I've verified that it is being decrypted and presented as expected. Do you have any other examples I can try?

edit: Facebook appears fine for me as well.

[jochristian]

Yes, that is what the guy at support told me as well.. He could not reproduce it.
Could it be related to which cdn server that you connect to? I'm in Europe. As far as I know FB have servers all over the world.
Also we use IPv6 in our office. Since Facebook supports IPv6 that is what we use when we go that webpage.

/Jo Christian

[jochristian]

Hmm, no..

I managed to reproduce this issue on my PA-200 running 7.0.2 as well.
But I do not have ipv6 at home. So this issue is NOT related to ipv6 it seems.

But I do get an certificate error on "fbcdn-profile-a.akamaihd.net".

/Jo Christian

[Gertjan-HFG]

mmmh interesting i am in europe to.

In my screenshot there is the actual IP  from the serving host.

@ITCMPHC do you have te opportunity to put that adres en the domain name from my link in your pc's hostfile.

And then check the link?

[ITCMPHC]

@Gertjan-HFG

Digging deeper I found that I had a test decrypt policy applied to me with the "Block sessions with untrusted issuers" unchecked. If I recall correctly, I had made this policy to test against a very similiar bug in 6.x. Removing that policy from my traffic now results in the same Untrusted error that you get. I tried importing both the Baltimore root cert and the Verizon intermediate manually, but it still results in the same error. Sorry for the confusion on this one. I'll open a ticket with support as well.

[Gertjan-HFG]

@ITCMPHC OK thx for your update.

[Sully]

I am having the same issues and have opened a case.  Waiting for TAC to lab up.