Performance Degradation for SSL Decryption

[Farzana]

Hello,

The issues we are experiencing are with SSL decrypt. When this setting is enabled we are experiencing significantly degraded internet performance.

We understand that this would have an overhead but the current overhead makes it almost unusable. The symptoms are worse on pages such as youtube.com due to the ads.

We have tested with SSL decrypt disabled and performance is as expected however as soon SSL decrypt is enabled an significant performance decrease is notice.

In the hope to resolve we have tested on the following versions however the issue is present on both versions.

• Reproduced issue on PAN-OS 7.1.8
• Reproduced issue on PAN-OS 8.0.12

Any advice would be appreciated.

[BPry]

@Farzana,

What is the device utilization when you're seeing this and what platform are you doing this on. The only time I've really seen issues with enabling decryption like what you're seeing is when the firewall is hitting its limits with the additional overhead of SSL Decryption being enabled. 

[LukeBullimore]

Hi @Farzana,

As a follow up to @BPry's message, you can use the command "show session all filter ssl-decrypt yes count yes" to see the number of current decrypted sessions and compare this with your firewall models maximum value. In combination to this, you should use the command "show running resource-monitor" to monitor the dataplane utilization if you notice "func_ssl_proxy_proc" hogging all the CPU, decryption may be maxing out your box and you would either need to limit what you're decrypting if you want to continue using your current hardware - or otherwise consider an upgrade.

Cheers,

Luke.

[aayoung]

Try disabling "ECDHE" in your decryption profile for your decryption policy, or figure out how you can streamline your decryption policy. You will lose Perfect Forward Secrecy ability though.  Like a few other have indicated you are probably pushing the limit on you r platforms decrypt seesions.

[FarzanaMustafa]

Hi @LukeBullimore

We are using PA-3060 and decrpyting most traffic due to network requirement. I ran the commands as you suggested but could not locate func_ssl-proxy_proc. When ran the command > show counter global filter packet-filter yes delta yes 

this is what we see below. Any idea if SSL decryption is causing the performance issue?

st in ssl proxy
proxy_url_category_unknown 10 0 info proxy pktproc Number of sessions checked by proxy with unknown url category
proxy_wait_pkt_drop 1088 3 drop proxy pktproc The number of packets get dropped because of waiting status in ssl proxy
proxy_l2hdr_extended 28322 100 info proxy pktproc Layer 2 header extended than original length
ssl_cert_cache_miss 9 0 info ssl pktproc Number of SSL certificate cache miss
ssl_cert_verify 39 0 info ssl pktproc Number of SSL certificates that need to do verify
ssl_rsa_key_cache_hit 9 0 info ssl pktproc Number of SSL RSA key cache hit
ssl_client_sess_ticket 55 0 info ssl pktproc Number of ssl session with client sess ticket ext
ssl_extended_master_secret 5 0 info ssl pktproc Number of ssl session created using extended master extension
url_db_request 13 0 info url pktproc Number of URL database request
zip_process 21 0 info zip resource The outstanding zip processes
zip_process_total 21 0 info zip pktproc The total number of zip engine decompress process
zip_process_stop 4 0 info zip pktproc The number of zip decompress process stops lack of output buffer
zip_hw_in 84805 300 info zip pktproc The total input data size to hardware zip engine
zip_hw_out 276073 976 info zip pktproc The total output data size from hardware zip engine

[LukeBullimore]

Hi @FarzanaMustafa,

Apologies for the confusion. The ssl_proxy_proc counters I was referring to can be found in the dp-monitor log. (less dp-log dp-monitor.log)

If you then have any access to any resources such as PANTS or AutoAssistant then you can correlate these counters to build graphs and compare this to the timestamps of when you notice your issue. 

[Brandon_Wertz]

What is your Internet circuit or the BW you're trying to push through the FW?

How many current sessions is the 3060 processing?

Can you estimate how many of these sessions are SSL?

How much of the total throughput is SSL traffic?

[FarzanaMustafa]

Hi all,

Just wanted to let you know that PA TAC team has asisted us in resolving the issue.

Browsing speed is now back to normal.

Device >Session> Decryption Settings, select Certificate Revocation Checking 
Uncheck CRL and OCSP.
Commit.

[LukeBullimore]

Hey @FarzanaMustafa

Interesting, glad you got to the bottom of it; although I believe these options are in fact unchecked by default.

[Remo]

@LukeBullimore is right, these options are unchecked by default but are recommended to check for a secure tls proxy configurarion:https://www.paloaltonetworks.com/documentation/80/best-practices/best-practices-decryption/decryptio... (and Paloalto also recommends to be careful with them as they will have a performance impact). But because of the (really) extreme performance degradation primarily the OCSP option is usless - unless you can live with unhappy users and a lot of complaints of them...

Only with the CRL option the performance is good, thats why we are only using this. Without any of them you accept the risk that users connect to websites with revoked certificates (for example if a cert is stolen and used by attackers even after the actual owner revoked the stolen cert)

[MP18]

i also checked my PA agree it is uncheck by default.

[MP18]

Also can you please confirm if we can enable the e CRL option and will have no impact on the performance?

[BPry]

@MP18,

Enabling either the CRL or OCSP options to check certificate status will have an effect on performance. CRL is much easier on the firewall and has a minimal impact, most people can enable this without a huge performance impact; while OCSP has a pretty massive performance hit and would really only be recommended if you need it for regulatory reasons. 

[MP18]

Thanks for confirming that.

Good to learn from you

[Ramakrishnan]

I read the entire thread. I wanted to fact the VM size, considering this as design principal that, SSL decryption consume high CPU, is there any SSL decryption sessions Vs throughput which could help to choose VM size(however due to this flexi consumption no more VM 3/5/700 but wanted to factor with rough estimate)