SSL Decryption Issue on PA-520 Running PAN-OS 12.1.7-h3

[M.Mohsen906766]

We are experiencing an issue with SSL decryption on a PA-520 running PAN-OS 12.1.7-h3.

After enabling decryption, the firewall operates normally for approximately three hours. However, over time, the number of decryption failures gradually increases until all new connections are rejected. Disabling decryption immediately restores normal operation and resolves the issue.

We observe multiple decryption-related errors, including:

• Certificate verify failed
• TLS handshake failure
• Invalid server certificate

The issue affects well-known websites such as google.com and youtube.com, which are known to have valid certificates. This suggests that the problem is not related to the remote servers' certificates.

Additionally, the affected connections are using TLS 1.3, so I believe obtaining the server certificate through packet captures may not be feasible.

Could you please advise on the best approach to investigate this issue further? Any recommendations regarding diagnostics, debugging, or known causes would be greatly appreciated.

Certificate verify failed
TLS Handshake failure
Invalid server certificate

The error appears for a well known sites like google.com, youtube.com. which for sure doesn't have a certificate problem.

The connections are using TLSv3, so i believe extracting  the certificate through packet capture will not work.

Any suggestion for investigation will be much appreciated. 

[JayGolf]

Hi @M.Mohsen906766 ,

I’d look more at a dataplane/decryption resource issue than the certificates themselves.

The main thing that stands out is that decryption works for a few hours, then failures gradually increase until new sessions stop working. When the issue starts happening, can you check: "debug dataplane pool statistics" Specifically, look at the proxy-flexmem-pool and compare it to when decryption is healthy.

I’d also check: "show counter global" and look for any resource or malloc-related errors. If the proxy pool is steadily being depleted as the failures increase, that would point much more toward a firewall-side resource issue.

[M.Mohsen906766]

Hello,

Thanks a lot for your help, I checked show counter globa & debug dataplane pool statistics and cannot see a problem. Can you help please to double check both outputs attached.

[BPry]

@M.Mohsen906766,

You're probably not going to find many people willing to download attachments on the forum. I would highly recommend opening a case and working through this with support since they can actually review all of your logs. Don't be surprised to see them request you update (there's a couple of decryption related issues addressed in later releases) and to have them ask for a troubleshooting session while you are actively experiencing the issue. Since this would be something you can leave enabled on a single host for troubleshooting, this should be relatively easy for them once you get through the first line. 

[Brandon_Wertz]

@M.Mohsen906766 -- Along with what @BPry  said I'd suggest looking at the release notes for 12.1.7+ through 12.1.10.  Looking for any SSL decryption / system resource bugs.  It's possible you're running into a bug that's know and has a planned release.  Aside from that, I would also suggest opening a ticket to have TAC more properly investigate this issue.

[PhilBoe]

Hi, we had a similar issue with one of our customers. Using 12.1.7-h2 all ssl decryption stopped showing "invalid server certificate".

Today we installed 12.1.10 and rebooted the firewall.

Issue was gone.

Now, we don't know if it was the reboot or software upgrade that solved it but we do think some kind of memory leak triggered it as it started during normal operation conditions.

[MP18]

@M.Mohsen906766 Also if device is in HA doing failover helps and if it is single device doing reboot will clear the memory if issue is causing memory issue in firewall.

I have seen the memory issues while doing decryption on the firewall and above steps fixed it.

As everyone mentioned open TAC case.

Also, Reboot is Temp fix of the issue.

Regards

[OtakarKlier]

Hello,

I agree with @Brandon_Wertz , there is a SSL decryption bug that 'was' fixed but has regressed and its with engineering. This is especially true if you are running in FIPS mode.

Regards,