topic Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted in General Topics

Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

gudmundur — Fri, 18 Sep 2015 19:19:27 GMT

We have noticed this with two customers and on our own PA's , all of these are PA3020's in a HA a/s setup

SSL decrypted outbound traffic hangs intermittently for a few minutes and then it starts to pass through again.

This happens both with 7.0.1 and 7.0.2

anyone seen this issue as well ?

kinda hard to work with support on this since it's intermittent

regards

Gudmundur

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

ITCMPHC — Mon, 21 Sep 2015 14:43:13 GMT

Yes, we have a ticket open regarding this right now. What is happening is the FPTCP buffer is filling up and not releasing like it should. Once this happens, the SSL Proxy engine drops packets until the buffer finally clears. We don't yet know what the true cause of the buffer filling up is.

You can verify you are experiencing the same bug by SSHing into the firewall and issueing the command "debug dataplane pool statistics". There you will see the line "FPTCP segs" (###): ###/###

The 3rd number is the max amount of buffer and the 2nd number is how much is left. When the issue occurs, you will notice the 2nd number stuck at 1. Eventually it will release on its own and traffic will flow again. Alternatively if you have an HA pair, you can fail over and it will immediately resolve.

edit: our ticket is 00379855 and the bug id is 84781, if you'd like to reference those with support.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

bgmncwj — Tue, 22 Sep 2015 15:32:25 GMT

I'm seeing the same thing here on 7.0.1, I was hoping 7.0.2 would've fixed it but I guess not. I just opened a ticket with PA and referenced the ticket @ITCMPHC has. Hopefully it'll be fixed soon, it's quite annoying when it happens.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

ITCMPHC — Wed, 23 Sep 2015 15:01:05 GMT

I've been told this will be fixed in 7.0.3 which is tentatively scheduled for Oct 19.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

gudmundur — Sat, 26 Sep 2015 11:34:08 GMT

Thanks for the update guys

I hope that 7.0.3 will fix this

regards

Gudmundur

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

bgmncwj — Tue, 20 Oct 2015 15:26:28 GMT

We took the plunge and upgraded to 7.0.3 and it doesn't look like this was fixed, at least in our case. I reopened our ticket with PA, we'll see what they say.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

ITCMPHC — Wed, 21 Oct 2015 14:57:39 GMT

I upgraded last night and am still having the issue. I've reported it to support as well.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

dannon — Wed, 21 Oct 2015 19:11:30 GMT

We are also having the same issue.

I've got an open TAC case with log files and information. Hopefully this will be fix ASAP. We rely heavily on SSL decryption.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

bgmncwj — Wed, 21 Oct 2015 19:14:16 GMT

Yeah this is really disappointing especially after they confirmed multiple times that it was fixed in 7.0.3. I don't really want to wait another month+ for the next software release for yet another fix. Sadly there's some other fixes we needed in 7.0.3 that stop us from going back.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

wh4t3ver — Wed, 21 Oct 2015 20:37:45 GMT

I'm still having this same issue as well.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

dannon — Thu, 22 Oct 2015 15:59:18 GMT

Just as an addition, I had a "dataplane under severe load" happen again this morning. System reasources applet on the Dashboard didn't register much, but the logs showed the CPUs were at 100% for well over 20 mins. This is also with 7.0.3, which should have fixed it. Spent 45 mins on thep hone with support and they gathered the log files to look at causes, but did say that our box should not behave this way given the size of our organization.

As we just upgraded from a 3020 to a 5050 to fix this issue, we are NOT happy about this. Palo has really slipped with the 7.0.x code. If you spend $100,000, you should NOT expect these kind of issues.

Rant over.

Dannon

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

ITCMPHC — Tue, 27 Oct 2015 16:57:39 GMT

Support says this is now being "tracked" as part of 7.0.4 which is slated for a December release.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

ulti — Tue, 27 Oct 2015 17:19:45 GMT

Palo's QA has really taken a back seat recently...

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

dannon — Tue, 27 Oct 2015 18:37:44 GMT

Yeah, I am disappointed by this bug. I installed 7.0.0 initially and ran for a month or 2 before seeing 7.0.1. I also noticed at that time that 7.0.0 had been pulled. Installed 7.0.1 and ran into issues with the SSL. Saw 7.0.2 but read about lots of bugs, some new with SSL and dataplane issues too. Was told by support to NOT go to 7.0.2 and wait for 7.0.3. Finally went to that and still have the SSL issues.

We are very disappointed here with what's been going on. SSL decryption is a big thing, and to have it borked up is poor QA for sure.

On a slightly better note, support got back to me with a temporary work-around to our issue. Basically, they had me create a new DOS protection profile and policy which only applies to in/out traffic on service-https. They said someting about it fixing the issue in testing and for other affected customers. Note this is a temp-fix until they get a proper update rolled out.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

wh4t3ver — Tue, 27 Oct 2015 20:34:25 GMT

Dannon, would you please post more details on the temp fix?

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

Retired Member — Wed, 28 Oct 2015 12:04:02 GMT

Hi,

Can you please provide more info over the work around.

thx.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

Mike_Vietorisz — Wed, 28 Oct 2015 23:48:29 GMT

Just chiming in here, we've had loads of problems with SSL Decryption (among other issues) since upgrading to 7.0.X. Not only the 'intermittent' dropout exactly described here, but increasing incompatibilities with various websites and applications running over HTTPS as well. Our decryption 'exception' list is growing rather large.

Very disappointed this wasn't fixed in 7.0.3.

I also find it sadly ironic that Palo Alto's own support portal file upload tool doesn't work when SSL Decryption is turned on...

Agree with the poster above... PA QA team really dropped the ball with the 7.0 release.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

ulti — Thu, 29 Oct 2015 15:28:30 GMT

is anyone at Palo reading these posts? Are they going to chime in and acknowledge this or are they just lurking and not supporting their paying customes wishing to provide feedback?

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

dannon — Thu, 29 Oct 2015 17:02:26 GMT

Well, Palo's temp fix wasn't really a fix for us. We had to disable it because it caused some HTTPS websites to not come up: outlook.com , youtube, etc. As soon as I disabled the fix, they came up without error. This is still with ssl decryption turned on.

If anyone wants to know what the fix is:

Hi Daniel,

It was nice talking to you earlier. In our session we discssued that engineering is actively working on the issue to resolve it.

In the meantime, there is a workaround in place. We will like to apply it at your end and see if that resolves the issue. We will configure DoS policy with aggregate profile on port 443.

1. Create DoS Profile Objects -> Security Profiles -> DoS Protection

<entry name="tac.case.00393841">
<flood>
<tcp-syn>
<syn-cookies>
<block>
<duration>10</duration>
</block>
<alarm-rate>1000000</alarm-rate>
<activate-rate>0</activate-rate>
<maximal-rate>1000000</maximal-rate>
</syn-cookies>
<enable>yes</enable>
</tcp-syn>

2. Create DoS policy, Policies -> DoS Protection

<entry name="tac.case.00393841">
<from>
<zone>
<member>Trust</member>
</zone>
</from>
<to>
<zone>
<member>Untrust</member>
</zone>
</to>
<protection>
<aggregate>
<profile>tac.case.00393841</profile>
</aggregate>
</protection>
<source>
<member>any</member>
</source>
<destination>
<member>any</member>
</destination>
<source-user>
<member>any</member>
</source-user>
<service>
<member>service-https</member>
</service>
<log-setting>Any_profile_If_Available</log-setting>
<action>
<protect/>
</action>
</entry>
</rules>
</dos>

Please do let us know if this resolves your issue. We will wait for your response to proceed. Thank you.

Re: Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

bbilut — Thu, 29 Oct 2015 17:11:25 GMT

Same here.. I opened a ticket 2 days ago. I'm running 7.0.3 on 5050 hardware active/standby.

Support is wanting me to take a memory dump during the condition when the FPTCP segs are depleted. I've had to disable SSL decryption because I was getting these hangs about every 30 minutes and all SSL traffic would hang for 5 minutes.