Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

[gudmundur]

Hi

We have noticed this with two customers and on our own PA's , all of these are PA3020's in a HA a/s setup 

SSL decrypted outbound traffic hangs intermittently for a few minutes and then it starts to pass through again.

This happens both with 7.0.1 and 7.0.2 

anyone seen this issue as well ?

kinda hard to work with support on this since it's intermittent 

regards

Gudmundur

[ulti]

starting asking your SE to putinternal pressure on this; I fear the admins aren't alerting the right internal Palo folks on the importance of this

[Brandon_Wertz]

Fortunately we didn't upgrade to 7.X on any of our firewalls (thanks to the community's responses on how much it sucks)...ok it doesn't suck but it's been riddled with problems.

To me it seems this issue has been big enough that an off-cycle patch should have done and Palo should have spent engineering time 24x7 until these issues had been fixed.  The cost of support 6-figures+ to have this large of a problem with a "wait for our patch cycle" mentality blows my mind.

[bbilut]

I asked for an update on my ticket and received the following back. I'd like to see the fix sooner, but it is what it is I guess.

Working with the engineering team I have been able to confirm the root cause of the issue is the result of Bug 85091. The resolution for this issue is currently scheduled for release in the 6.1.8 and 7.0.4 software update.

The new software versions are expected to be released the Week of November 16, 2015 for 6.1.8 and the Week of December 7, 2015 for 7.0.4.

[bgmncwj]

I wonder how big a problem has to be before they patch it out of the normal release cycle?

[bgmncwj]

Is anybody running 6.1.x having this problem? I hadn't heard of it occuring in the 6.x line until the PA response stating it'll be fixed in 6.1.8. I'm concerned that once again they've missed the root cause.

[bbilut]

I found that weird as well. I was running 6.1.7 with no issues until I upgraded to 7.0.3. At least I was not aware of any issues... maybe it's more pronounced in 7.0.x??

[nhogan]

Hi all- I’m Nathan, with Support Management at Palo Alto Networks. As was posted earlier, we’re sorry for frustration caused by this issue. The support response details posted above are correct. 6.1.8 is planned for release later this week (16-Nov) and 7.04 will be available during the week of 21-Dec (please note the date, as it is different than what was posted earlier). Both of these releases resolve this issue.

There may be additional options available, but support would prefer to discuss these individually to ensure they meet your individual needs and timeliness. I’ve reached out to some of the members on this discussion via phone/email to provide some options related to this issue. While we recommend upgrading when the releases become available, there may also be a workaround option for those that require an immediate fix. The workaround has been tested by our QA group, which is why we would like to have a live debug session for those that are having trouble implementing the workaround. For those I’ve contacted that need to pursue this path, please get back to me and I will make sure you get help from Support to discuss all options.

Note: if there are others experiencing this issue who have not created a support case, please contact Palo Alto Networks support (https://www.paloaltonetworks.com/company/contact-us.html). We’re happy to help get this resolved for you.

Thanks,

Nathan

[Gertjan-HFG]

What is the workaround? Have a call open, but maybe the community is faster.

[bgmncwj]

There really isn't (from what I've been told and discovered after implementing the work-around). They're recommending people configure some DoS rules against SSL. This will extend the interval between the issue happening and traffic halting but not fix it.

[bbilut]

Yes, it hasn't fixed the problem for me, but it's made the freezes much less often to several pay day rather than every 45 minutes as I was seeing prior. I've been able to turn decryption back on, and my users are dealing with the pain right now.

[bbilut]

This is what they did for me.

[Gertjan-HFG]

Thanks Guys, will try the DOS rule.

[bgmncwj]

For reference here's the work-around I was provided:

set profiles dos-protection profiledos type aggregate flood tcp-syn enable yes syn-cookies activate-rate 0 alarm-rate 1000000 maximal-rate 1000000 block duration 100
set rulebase dos rules dosrule2 service service-https source 192.168.1.10 destination 1.1.1.2 action protect
set rulebase dos rules dosrule2 from zone L3-Trust
set rulebase dos rules dosrule2 to zone L3-Untrust
set rulebase dos rules dosrule2 protection aggregate profile profiledos

Everybody running into this problem should still open a case though to show how many people are being affected.

[bbilut]

They had me create a reverse rule as well. I did mine based on zones rather than networks IP

trusted  ---> untrusted

untrusted ---> trusted

[tpatrick]

Can anyone who has updated to 6.1.8 verify if the issue has been resolved by the update?  Support claims the issue is resolved in both 6.1.8 and 7.0.4 so hopefully if it's resolved in 6.1.8 we can assume it will also be resolved in the upcoming 7.0.4 release.  

We're wanting to roll out 7.0.4 shortly after the release but only if we can verify the issue is resolved.