Performance Degradation for SSL Decryption

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Performance Degradation for SSL Decryption

L4 Transporter

Hello,

 

The issues we are experiencing are with SSL decrypt. When this setting is enabled we are experiencing significantly degraded internet performance.

We understand that this would have an overhead but the current overhead makes it almost unusable. The symptoms are worse on pages such as youtube.com due to the ads.

We have tested with SSL decrypt disabled and performance is as expected however as soon SSL decrypt is enabled an significant performance decrease is notice.

In the hope to resolve we have tested on the following versions however the issue is present on both versions.

    • Reproduced issue on PAN-OS 7.1.8
    • Reproduced issue on PAN-OS 8.0.12

Any advice would be appreciated.

14 REPLIES 14

Cyber Elite
Cyber Elite

@Farzana,

What is the device utilization when you're seeing this and what platform are you doing this on. The only time I've really seen issues with enabling decryption like what you're seeing is when the firewall is hitting its limits with the additional overhead of SSL Decryption being enabled. 

L5 Sessionator

Hi @Farzana,

 

As a follow up to @BPry's message, you can use the command "show session all filter ssl-decrypt yes count yes" to see the number of current decrypted sessions and compare this with your firewall models maximum value. In combination to this, you should use the command "show running resource-monitor" to monitor the dataplane utilization if you notice "func_ssl_proxy_proc" hogging all the CPU, decryption may be maxing out your box and you would either need to limit what you're decrypting if you want to continue using your current hardware - or otherwise consider an upgrade.

 

Cheers,

Luke.

L1 Bithead

Try disabling "ECDHE" in your decryption profile for your decryption policy, or figure out how you can streamline your decryption policy. You will lose Perfect Forward Secrecy ability though.  Like a few other have indicated you are probably pushing the limit on you r platforms decrypt seesions.

Hi @LukeBullimore

 

We are using PA-3060 and decrpyting most traffic due to network requirement. I ran the commands as you suggested but could not locate func_ssl-proxy_proc. When ran the command > show counter global filter packet-filter yes delta yes 

this is what we see below. Any idea if SSL decryption is causing the performance issue?

 

st in ssl proxy
proxy_url_category_unknown 10 0 info proxy pktproc Number of sessions checked by proxy with unknown url category
proxy_wait_pkt_drop 1088 3 drop proxy pktproc The number of packets get dropped because of waiting status in ssl proxy
proxy_l2hdr_extended 28322 100 info proxy pktproc Layer 2 header extended than original length
ssl_cert_cache_miss 9 0 info ssl pktproc Number of SSL certificate cache miss
ssl_cert_verify 39 0 info ssl pktproc Number of SSL certificates that need to do verify
ssl_rsa_key_cache_hit 9 0 info ssl pktproc Number of SSL RSA key cache hit
ssl_client_sess_ticket 55 0 info ssl pktproc Number of ssl session with client sess ticket ext
ssl_extended_master_secret 5 0 info ssl pktproc Number of ssl session created using extended master extension
url_db_request 13 0 info url pktproc Number of URL database request
zip_process 21 0 info zip resource The outstanding zip processes
zip_process_total 21 0 info zip pktproc The total number of zip engine decompress process
zip_process_stop 4 0 info zip pktproc The number of zip decompress process stops lack of output buffer
zip_hw_in 84805 300 info zip pktproc The total input data size to hardware zip engine
zip_hw_out 276073 976 info zip pktproc The total output data size from hardware zip engine

Hi @FarzanaMustafa,

 

Apologies for the confusion. The ssl_proxy_proc counters I was referring to can be found in the dp-monitor log. (less dp-log dp-monitor.log)

 

If you then have any access to any resources such as PANTS or AutoAssistant then you can correlate these counters to build graphs and compare this to the timestamps of when you notice your issue. 

 

What is your Internet circuit or the BW you're trying to push through the FW?

 

How many current sessions is the 3060 processing?

 

Can you estimate how many of these sessions are SSL?

 

How much of the total throughput is SSL traffic?

L4 Transporter

Hi all,

 

Just wanted to let you know that PA TAC team has asisted us in resolving the issue.

Browsing speed is now back to normal.

Device >Session> Decryption Settings, select Certificate Revocation Checking 
Uncheck CRL and OCSP.
Commit.

Hey @FarzanaMustafa

 

Interesting, glad you got to the bottom of it; although I believe these options are in fact unchecked by default.

@LukeBullimore is right, these options are unchecked by default but are recommended to check for a secure tls proxy configurarion:https://www.paloaltonetworks.com/documentation/80/best-practices/best-practices-decryption/decryptio... (and Paloalto also recommends to be careful with them as they will have a performance impact). But because of the (really) extreme performance degradation primarily the OCSP option is usless - unless you can live with unhappy users and a lot of complaints of them...

Only with the CRL option the performance is good, thats why we are only using this. Without any of them you accept the risk that users connect to websites with revoked certificates (for example if a cert is stolen and used by attackers even after the actual owner revoked the stolen cert)

i also checked my PA agree it is uncheck by default.

MP

Help the community: Like helpful comments and mark solutions.

Also can you please confirm if we can enable the e CRL option and will have no impact on the performance?

MP

Help the community: Like helpful comments and mark solutions.

@MP18,

Enabling either the CRL or OCSP options to check certificate status will have an effect on performance. CRL is much easier on the firewall and has a minimal impact, most people can enable this without a huge performance impact; while OCSP has a pretty massive performance hit and would really only be recommended if you need it for regulatory reasons. 

Thanks for confirming that.

Good to learn from you

MP

Help the community: Like helpful comments and mark solutions.

L2 Linker

I read the entire thread. I wanted to fact the VM size, considering this as design principal that, SSL decryption consume high CPU, is there any SSL decryption sessions Vs throughput which could help to choose VM size(however due to this flexi consumption no more VM 3/5/700 but wanted to factor with rough estimate)   

  • 18499 Views
  • 14 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!