Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

[gudmundur]

Hi

We have noticed this with two customers and on our own PA's , all of these are PA3020's in a HA a/s setup 

SSL decrypted outbound traffic hangs intermittently for a few minutes and then it starts to pass through again.

This happens both with 7.0.1 and 7.0.2 

anyone seen this issue as well ?

kinda hard to work with support on this since it's intermittent 

regards

Gudmundur

[Gertjan-HFG]

Hi,

Can you please provide more info over the work around.

thx.

[Mike_Vietorisz]

Just chiming in here, we've had loads of problems with SSL Decryption (among other issues) since upgrading to 7.0.X. Not only the 'intermittent' dropout exactly described here, but increasing incompatibilities with various websites and applications running over HTTPS as well. Our decryption 'exception' list is growing rather large.

Very disappointed this wasn't fixed in 7.0.3.

I also find it sadly ironic that Palo Alto's own support portal file upload tool doesn't work when SSL Decryption is turned on...

Agree with the poster above... PA QA team really dropped the ball with the 7.0 release.

[ulti]

is anyone at Palo reading these posts? Are they going to chime in and acknowledge this or are they just lurking and not supporting their paying customes wishing to provide feedback?

[dannon]

Well, Palo's temp fix wasn't really a fix for us.  We had to disable it because it caused some HTTPS websites to not come up: outlook.com , youtube, etc.  As soon as I disabled the fix, they came up without error.  This is still with ssl decryption turned on.

If anyone wants to know what the fix is:

Hi Daniel,

It was nice talking to you earlier. In our session we discssued that engineering is actively working on the issue to resolve it. 

In the meantime, there is a workaround in place. We will like to apply it at your end and see if that resolves the issue. We will configure DoS policy with aggregate profile on port 443.

1. Create DoS Profile Objects -> Security Profiles -> DoS Protection

<entry name="tac.case.00393841">
<flood>
<tcp-syn>
<syn-cookies>
<block>
<duration>10</duration>
</block>
<alarm-rate>1000000</alarm-rate>
<activate-rate>0</activate-rate>
<maximal-rate>1000000</maximal-rate>
</syn-cookies>
<enable>yes</enable>
</tcp-syn> 

2. Create DoS policy, Policies -> DoS Protection

<entry name="tac.case.00393841">
<from>
<zone>
<member>Trust</member>
</zone>
</from>
<to>
<zone>
<member>Untrust</member>
</zone>
</to>
<protection>
<aggregate>
<profile>tac.case.00393841</profile>
</aggregate>
</protection>
<source>
<member>any</member>
</source>
<destination>
<member>any</member>
</destination>
<source-user>
<member>any</member>
</source-user>
<service>
<member>service-https</member>
</service>
<log-setting>Any_profile_If_Available</log-setting>
<action>
<protect/>
</action>
</entry>
</rules>
</dos>

Please do let us know if this resolves your issue. We will wait for your response to proceed. Thank you.

[bbilut]

Same here.. I opened a ticket 2 days ago. I'm running 7.0.3 on 5050 hardware active/standby.

Support is wanting me to take a memory dump during the condition when the FPTCP segs are depleted. I've had to disable SSL decryption because I was getting these hangs about every 30 minutes and all SSL traffic would hang for 5 minutes.

[Mike_Vietorisz]

Can either the original poster or someone from PA take away the 'Solved' marking for this thread? This issue is not resolved.

[jochristian]

Hi,

Do you have the case number for this problem and the bug id?

/Jo Christian

[sokonta]

Hi Guys,

Just checking if anyone here has found a work around on this issue, We are very close to turning off SSL Decryption until there is a permanent fix.  I will hold off until i hear from someone.

Thanks

Sol.

[bbilut]

I worked with support this morning and turned my decryption back on so we can get a memory dump during the event (which we did), and then I turned decryption back off because for me it causes a 5-8 minute outage every 45 minutes. They way I see it is the only solution I'm hear at this time is to backrev to 6.1.x or turn off decryption. I don't like either solution, but I don't feel like going back to 6.1.7 at this point.

[bgmncwj]

I did call out this thead in my ticket to at least make them aware of the scope of the problem if they weren't already.

Now they want me to turn back on SSL decryption and break a production network to capture some more log details. Don't they have labs for this sort of thing? I can't imagine all of our configurations are similar enough to cause this problem but they can't reproduce it internally.

[bbilut]

I know how you feel I had to do the same thing. I turned decryption back on overnight, and I setup a early morning time to do the captures they wanted and quickly turned decryption off again. They told me they opened bug 86948 after I took my captures.

[Retired Member]

Hi all – wanted to chime in thank all of you affected for your patience on this issue. We are sorry this is creating frustration. The team is very focused on getting it resolved. As we try to target and resolve the issue, the more insight we can gain from different traffic flows, the better. So – if you’re experiencing issues with this and haven’t contacted support yet, please do so. It will help us help you.

Thanks,

- Jeff

[bbilut]

Is there any new news on this issue. My students are starting to figure out that decryption is off and are starting to run wild. I'll really don't want to roll back to 6.1.x, but I may be forced to do that. <SIGH>

[bgmncwj]

So far my case (381368) is still in 'Engineering Escalation' and was last touched on 11/5. I haven't heard anything about a expected resolution date or release.

[bbilut]

Yea my case has the same status.