topic Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1) in General Topics

PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Retired Member — Sat, 29 Aug 2015 21:27:07 GMT

Yesterday i upgraded my pa vm-100 from panos-7.01 to 7.02.

After that facebook stopped working with SSL decryption on.

After some testing and troubleshooting this seems to be the problem.

The problem is that some akamai domains that facebook uses gives me an palo alto certificate untrusted page.

for example this domain: https://fbcdn-profile-a.akamaihd.net

The strange thing is all the certificates used by this domain are already in de PA trusted cert auth list.

Just to be sure i downloaded the certs and added them manually to the PA, but no difference.

After spending 2 hours debugging en trying to get it work,

off course i can exclude those domains from decryption or or let the PA ingnore untrusted certs but thats not the way to do it. i downgraded to panos 7.0.1 and the untrusted cert problem dissapeared.

Are more people having this issue? i think there are more sites that stop working after the upgrade.

Does anyone found a solution?

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Lucky — Sun, 30 Aug 2015 07:55:47 GMT

Hi Gertjan,

Welcome to the community. That is an interesting display of the problem you have.

First things first, link you shared is https but apparently is not encrypted, there is no certificate attached to it? At least from my browser, I am lazy to check with curl. Do you have any other sample URLs that didn't work?

Secondly, if you found out there is a certificate - did you check on the issuer? It should be in the list of trusted certificates. I am thinking, if you can find a root certificate that signed those untrusted certificates, and install it to your device, afterwards it's signed certs will be trusted. Of course, this is only in the case you really are sure of aforementioned root certificate validity...

Did you try any of that? I haven't moved to 7.0.2 yet and I am not decrypting ATM but I would test it, provided you have valid URL 🙂

Regards

Luciano

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Retired Member — Sun, 30 Aug 2015 10:14:46 GMT

Here is a "problem" link from the palo alto facebook page:

https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfp1/v/t1.0-1/p160x160/11196343_862388790520989_3155649659031350584_n.png?oh=f0ab8f382afe87724078278c0b3ff300&oe=5674C658&__gda__=1449702963_90e15547f92014d9a68207fcd6546dcf

in panos 7.0.1 its works with SSL decryption after upgrade to 7.0.2 you get a certificate untrusted page from the Palo Alto,

Downgrade to 7.0.1 link works correct again.

I kown this page very well because we get montly request from our users who get this en then we add the root certificate on the Palo Alto and after that its working fine.

All the certificates used were already in the default PA trusted root, just to be sure I downloaded these certificates en installed them on the device but no difference. This is for 99% sure a panos-7.0.2 bug

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

pulukas — Sun, 30 Aug 2015 13:55:11 GMT

Thanks for the pretty complete testing. I agree this is likely a bug.

Did you open an official support ticket so this can be logged in the bug database and fixed?

These forums are informal community support. You do need an official ticket to get the bug report created.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

jochristian — Mon, 31 Aug 2015 06:24:43 GMT

Hi,

I'm seeing the same problem.
I have already opened an case with TAC on this.

/Jo Christian

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Retired Member — Mon, 31 Aug 2015 08:03:02 GMT

Ok, I did not open a case with TAC yet.

Did you have problems on other sites than facebook?

Please keep us posted!

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Lucky — Mon, 31 Aug 2015 08:06:51 GMT

Hi Gertjan,

if you have support account, it is better if you open the case as well and point out to this discussion to the TAC engineer that takes the case; more cases = more reports on the bug = bigger weigth on fixing it 🙂 They will note it and add it to the already opened bug report.

Regards

Luciano

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

jochristian — Mon, 31 Aug 2015 10:46:34 GMT

Hi,

So far we have only seen this on Facebook.
But we did not test many other webpages before we downgraded to 7.0.1 (that works fine).

If you open an support case on this please refer to casenumber: 00371068

/Jo Christian

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

ITCMPHC — Mon, 31 Aug 2015 14:14:58 GMT

Interesting as I too upgraded to 7.0.2 on my 3050s last night and I'm not seeing this issue on the page you linked. I've verified that it is being decrypted and presented as expected. Do you have any other examples I can try?

edit: Facebook appears fine for me as well.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

jochristian — Mon, 31 Aug 2015 15:40:20 GMT

Yes, that is what the guy at support told me as well.. He could not reproduce it.
Could it be related to which cdn server that you connect to? I'm in Europe. As far as I know FB have servers all over the world.
Also we use IPv6 in our office. Since Facebook supports IPv6 that is what we use when we go that webpage.

/Jo Christian

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

jochristian — Mon, 31 Aug 2015 15:52:30 GMT

Hmm, no..

I managed to reproduce this issue on my PA-200 running 7.0.2 as well.
But I do not have ipv6 at home. So this issue is NOT related to ipv6 it seems.

But I do get an certificate error on "fbcdn-profile-a.akamaihd.net".

/Jo Christian

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Retired Member — Mon, 31 Aug 2015 16:17:40 GMT

mmmh interesting i am in europe to.

In my screenshot there is the actual IP from the serving host.

@ITCMPHC do you have te opportunity to put that adres en the domain name from my link in your pc's hostfile.

And then check the link?

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

ITCMPHC — Mon, 31 Aug 2015 16:43:30 GMT

@Retired Member

Digging deeper I found that I had a test decrypt policy applied to me with the "Block sessions with untrusted issuers" unchecked. If I recall correctly, I had made this policy to test against a very similiar bug in 6.x. Removing that policy from my traffic now results in the same Untrusted error that you get. I tried importing both the Baltimore root cert and the Verizon intermediate manually, but it still results in the same error. Sorry for the confusion on this one. I'll open a ticket with support as well.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Retired Member — Tue, 01 Sep 2015 07:10:22 GMT

@ITCMPHC OK thx for your update.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Sully — Tue, 01 Sep 2015 18:28:20 GMT

I am having the same issues and have opened a case. Waiting for TAC to lab up.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

wh4t3ver — Tue, 01 Sep 2015 20:14:33 GMT

I had this same issue on my 5050 units and had to roll back to 7.0.1 to fix. This issue affected multiple sites including www.paloaltonetworks.com.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Retired Member — Thu, 03 Sep 2015 13:13:02 GMT

Any updates from the TAC cases?

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Sully — Fri, 04 Sep 2015 18:27:24 GMT

Long call with TAC. No resolution besides rolling back to 7.0.1. Scheduling a maintenance window to roll back. Maybe 7.0.2 will join it's 7.0.0 cousin.

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

jochristian — Mon, 07 Sep 2015 06:27:46 GMT

Hi,

Seems like TAC is struggling to find the problem here and also reproduce it.
Can those of you that have opened up cases on this share the case numbers? That way we can make sure that supports know that this problem is common for many users.

My casenumber is 00371068

/Jo Christian

Re: PAN-OS 7.0.2 SSL Decryption certficate untrust issues (No problem on 7.0.1)

Sully — Tue, 08 Sep 2015 12:09:35 GMT

Case 00372222

@Sully wrote:
Long call with TAC. No resolution besides rolling back to 7.0.1. Scheduling a maintenance window to roll back. Maybe 7.0.2 will join it's 7.0.0 cousin.