Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

Reply
Highlighted
L3 Networker

Just as an addition, I had a "dataplane under severe load" happen again this morning.  System reasources applet on the Dashboard didn't register much, but the logs showed the CPUs were at 100% for well over 20 mins.  This is also with 7.0.3, which should have fixed it.  Spent 45 mins on thep hone with support and they gathered the log files to look at causes, but did say that our box should not behave this way given the size of our organization.

 

As we just upgraded from a 3020 to a 5050 to fix this issue, we are NOT happy about this.  Palo has really slipped with the 7.0.x code.  If you spend $100,000, you should NOT expect these kind of issues.

 

Rant over.

Dannon

 

 

 

Highlighted
L2 Linker

Support says this is now being "tracked" as part of 7.0.4 which is slated for a December release.

Highlighted
L3 Networker

Palo's QA has really taken a back seat recently...

Highlighted
L3 Networker

Yeah, I am disappointed by this bug.  I installed 7.0.0 initially and ran for a month or 2 before seeing 7.0.1.  I also noticed at that time that 7.0.0 had been pulled.  Installed 7.0.1 and ran into issues with the SSL.  Saw 7.0.2 but read about lots of bugs, some new with SSL and dataplane issues too.  Was told by support to NOT go to 7.0.2 and wait for 7.0.3.  Finally went to that and still have the SSL issues.

 

We are very disappointed here with what's been going on.  SSL decryption is a big thing, and to have it borked up is poor QA for sure.

 

On a slightly better note, support got back to me with a temporary work-around to our issue.  Basically, they had me create a new DOS protection profile and policy which only applies to in/out traffic on service-https.  They said someting about it fixing the issue in testing and for other affected customers.  Note this is a temp-fix until they get a proper update rolled out.

 

 

Highlighted
L1 Bithead

Dannon, would you please post more details on the temp fix?

Highlighted
L3 Networker

Hi,

 

Can you please provide more info over the work around.

thx.

Highlighted
L1 Bithead

Just chiming in here, we've had loads of problems with SSL Decryption (among other issues) since upgrading to 7.0.X. Not only the 'intermittent' dropout exactly described here, but increasing incompatibilities with various websites and applications running over HTTPS as well. Our decryption 'exception' list is growing rather large.

 

Very disappointed this wasn't fixed in 7.0.3.

 

I also find it sadly ironic that Palo Alto's own support portal file upload tool doesn't work when SSL Decryption is turned on...

 

Agree with the poster above... PA QA team really dropped the ball with the 7.0 release.

Highlighted
L3 Networker

is anyone at Palo reading these posts? Are they going to chime in and acknowledge this or are they just lurking and not supporting their paying customes wishing to provide feedback?

Highlighted
L3 Networker

Well, Palo's temp fix wasn't really a fix for us.  We had to disable it because it caused some HTTPS websites to not come up: outlook.com , youtube, etc.  As soon as I disabled the fix, they came up without error.  This is still with ssl decryption turned on.

 

If anyone wants to know what the fix is:

 

Hi Daniel,

It was nice talking to you earlier. In our session we discssued that engineering is actively working on the issue to resolve it. 

In the meantime, there is a workaround in place. We will like to apply it at your end and see if that resolves the issue. We will configure DoS policy with aggregate profile on port 443.

1. Create DoS Profile Objects -> Security Profiles -> DoS Protection

<entry name="tac.case.00393841">
<flood>
<tcp-syn>
<syn-cookies>
<block>
<duration>10</duration>
</block>
<alarm-rate>1000000</alarm-rate>
<activate-rate>0</activate-rate>
<maximal-rate>1000000</maximal-rate>
</syn-cookies>
<enable>yes</enable>
</tcp-syn> 

2. Create DoS policy, Policies -> DoS Protection

<entry name="tac.case.00393841">
<from>
<zone>
<member>Trust</member>
</zone>
</from>
<to>
<zone>
<member>Untrust</member>
</zone>
</to>
<protection>
<aggregate>
<profile>tac.case.00393841</profile>
</aggregate>
</protection>
<source>
<member>any</member>
</source>
<destination>
<member>any</member>
</destination>
<source-user>
<member>any</member>
</source-user>
<service>
<member>service-https</member>
</service>
<log-setting>Any_profile_If_Available</log-setting>
<action>
<protect/>
</action>
</entry>
</rules>
</dos>

Please do let us know if this resolves your issue. We will wait for your response to proceed. Thank you.

Highlighted
L3 Networker

Same here.. I opened a ticket 2 days ago. I'm running 7.0.3 on 5050 hardware active/standby.

 

Support is wanting me to take a memory dump during the condition when the FPTCP segs are depleted. I've had to disable SSL decryption because I was getting these hangs about every 30 minutes and all SSL traffic would hang for 5 minutes.

-Brad
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!