Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Reply
Highlighted
L1 Bithead

Post 7.0x upgrade intermittend SSL traffic hangs when being decrypted

Hi

 

We have noticed this with two customers and on our own PA's , all of these are PA3020's in a HA a/s setup 

SSL decrypted outbound traffic hangs intermittently for a few minutes and then it starts to pass through again.

 

This happens both with 7.0.1 and 7.0.2 

 

anyone seen this issue as well ?

kinda hard to work with support on this since it's intermittent 

 

regards

Gudmundur

Tags (1)
Highlighted
L2 Linker

Yes, we have a ticket open regarding this right now. What is happening is the FPTCP buffer is filling up and not releasing like it should. Once this happens, the SSL Proxy engine drops packets until the buffer finally clears. We don't yet know what the true cause of the buffer filling up is.

 

You can verify you are experiencing the same bug by SSHing into the firewall and issueing the command "debug dataplane pool statistics". There you will see the line "FPTCP segs" (###): ###/### 

The 3rd number is the max amount of buffer and the 2nd number is how much is left. When the issue occurs, you will notice the 2nd number stuck at 1. Eventually it will release on its own and traffic will flow again. Alternatively if you have an HA pair, you can fail over and it will immediately resolve.

edit: our ticket is 00379855 and the bug id is 84781, if you'd like to reference those with support.

Highlighted
L2 Linker

I'm seeing the same thing here on 7.0.1, I was hoping 7.0.2 would've fixed it but I guess not. I just opened a ticket with PA and referenced the ticket @ITCMPHC has. Hopefully it'll be fixed soon, it's quite annoying when it happens.

Highlighted
L2 Linker

I've been told this will be fixed in 7.0.3 which is tentatively scheduled for Oct 19. 

Highlighted
L1 Bithead

Thanks for the update guys 

 

I hope that 7.0.3 will fix this 

 

regards

Gudmundur

Highlighted
L2 Linker

We took the plunge and upgraded to 7.0.3 and it doesn't look like this was fixed, at least in our case. I reopened our ticket with PA, we'll see what they say.

Highlighted
L2 Linker

I upgraded last night and am still having the issue. I've reported it to support as well.

Highlighted
L3 Networker

We are also having the same issue.

 

I've got an open TAC case with log files and information.  Hopefully this will be fix ASAP.  We rely heavily on SSL decryption.

Highlighted
L2 Linker

Yeah this is really disappointing especially after they confirmed multiple times that it was fixed in 7.0.3. I don't really want to wait another month+ for the next software release for yet another fix. Sadly there's some other fixes we needed in 7.0.3 that stop us from going back.

Highlighted
L1 Bithead

I'm still having this same issue as well.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!