Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

L1 Bithead

Hi

 

We have noticed this with two customers and on our own PA's , all of these are PA3020's in a HA a/s setup 

SSL decrypted outbound traffic hangs intermittently for a few minutes and then it starts to pass through again.

 

This happens both with 7.0.1 and 7.0.2 

 

anyone seen this issue as well ?

kinda hard to work with support on this since it's intermittent 

 

regards

Gudmundur

59 REPLIES 59

starting asking your SE to putinternal pressure on this; I fear the admins aren't alerting the right internal Palo folks on the importance of this

Fortunately we didn't upgrade to 7.X on any of our firewalls (thanks to the community's responses on how much it sucks)...ok it doesn't suck but it's been riddled with problems.

 

To me it seems this issue has been big enough that an off-cycle patch should have done and Palo should have spent engineering time 24x7 until these issues had been fixed.  The cost of support 6-figures+ to have this large of a problem with a "wait for our patch cycle" mentality blows my mind.

I asked for an update on my ticket and received the following back. I'd like to see the fix sooner, but it is what it is I guess.

 

 

Working with the engineering team I have been able to confirm the root cause of the issue is the result of Bug 85091. The resolution for this issue is currently scheduled for release in the 6.1.8 and 7.0.4 software update.

The new software versions are expected to be released the Week of November 16, 2015 for 6.1.8 and the Week of December 7, 2015 for 7.0.4.

-Brad

I wonder how big a problem has to be before they patch it out of the normal release cycle?

Is anybody running 6.1.x having this problem? I hadn't heard of it occuring in the 6.x line until the PA response stating it'll be fixed in 6.1.8. I'm concerned that once again they've missed the root cause.

I found that weird as well. I was running 6.1.7 with no issues until I upgraded to 7.0.3. At least I was not aware of any issues... maybe it's more pronounced in 7.0.x??

-Brad

L1 Bithead

Hi all- I’m Nathan, with Support Management at Palo Alto Networks. As was posted earlier, we’re sorry for frustration caused by this issue. The support response details posted above are correct. 6.1.8 is planned for release later this week (16-Nov) and 7.04 will be available during the week of 21-Dec (please note the date, as it is different than what was posted earlier). Both of these releases resolve this issue.

 

There may be additional options available, but support would prefer to discuss these individually to ensure they meet your individual needs and timeliness. I’ve reached out to some of the members on this discussion via phone/email to provide some options related to this issue. While we recommend upgrading when the releases become available, there may also be a workaround option for those that require an immediate fix. The workaround has been tested by our QA group, which is why we would like to have a live debug session for those that are having trouble implementing the workaround. For those I’ve contacted that need to pursue this path, please get back to me and I will make sure you get help from Support to discuss all options.

 

Note: if there are others experiencing this issue who have not created a support case, please contact Palo Alto Networks support (https://www.paloaltonetworks.com/company/contact-us.html). We’re happy to help get this resolved for you.

 

Thanks,

Nathan

What is the workaround? Have a call open, but maybe the community is faster.

There really isn't (from what I've been told and discovered after implementing the work-around). They're recommending people configure some DoS rules against SSL. This will extend the interval between the issue happening and traffic halting but not fix it.

Yes, it hasn't fixed the problem for me, but it's made the freezes much less often to several pay day rather than every 45 minutes as I was seeing prior. I've been able to turn decryption back on, and my users are dealing with the pain right now.

-Brad

This is what they did for me.

 

2015-12-07 12_54_26-nadmmdfpa5050-1.png

2015-12-07 12_54_58-Program Manager.png

-Brad

L3 Networker

Thanks Guys, will try the DOS rule.

For reference here's the work-around I was provided:

 

set profiles dos-protection profiledos type aggregate flood tcp-syn enable yes syn-cookies activate-rate 0 alarm-rate 1000000 maximal-rate 1000000 block duration 100
set rulebase dos rules dosrule2 service service-https source 192.168.1.10 destination 1.1.1.2 action protect
set rulebase dos rules dosrule2 from zone L3-Trust
set rulebase dos rules dosrule2 to zone L3-Untrust
set rulebase dos rules dosrule2 protection aggregate profile profiledos

 

Everybody running into this problem should still open a case though to show how many people are being affected.

They had me create a reverse rule as well. I did mine based on zones rather than networks IP

 

trusted  ---> untrusted

untrusted ---> trusted

-Brad

Can anyone who has updated to 6.1.8 verify if the issue has been resolved by the update?  Support claims the issue is resolved in both 6.1.8 and 7.0.4 so hopefully if it's resolved in 6.1.8 we can assume it will also be resolved in the upcoming 7.0.4 release.  

 

We're wanting to roll out 7.0.4 shortly after the release but only if we can verify the issue is resolved.  

  • 18437 Views
  • 59 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!