SYN ACK RST Loop on inside and SYN ACK only - source port re-use - hung session

Announcements

ATTENTION Customers, All Partners and Employees: The Customer Support Portal (CSP) will be undergoing maintenance and unavailable on Saturday, November 7, 2020, from 11 am to 11 pm PST. Please read our blog for more information.

Reply
L1 Bithead

SYN ACK RST Loop on inside and SYN ACK only - source port re-use - hung session

I have had this happen twice so far, I am wondering how others have solved for it.

 

Host A -> Palo Alto -> Host B

 

Host A, cycles through its source ports frequently - every couple of minutes.

Host B has long  or no TCP keep alive timers.

 

Host A:src port X connects to Host B on port 443.

Host A has an issue, reboots, lost power, app crashes, etc.  But it does NOT close the session.  Host B shows the connection in the netstat.  

 

Host A:src port X tries to connect to host B on port 443.  Host B responds with an ACK, this is per RFC.  That ACK is passed back to host A, Host A responds to the ACK with a RST.  The firewall drops the reset, and the connection is stuck.  Every time the host cycles through that source port it never clears.

 

Looking on the firewall on the A side, all I see is

Syn ->

<- ACK

RST->

For like 6 tries from the host on this source port, then is moves on until it gets back around to this port and it does it again and again.  No way to clear it.

 

On the B side, I see this

Syn ->

<- ACK

No reset, the firewall is eating the reset.

 

I tried putting a deny rule in with reset in both directions, but it only resets on the A side.

Clearing the session in the session table does not work.

 

I have tried to spoof reset packets and they are not getting through the firewall.

 

The only thing I have been able to do is have the server guy either clear the session or reboot it if they dont have the capabliity to clear a stale session.  Ask them to turn on keepalives to try to prevent, but this does not seem right.  

 

This site shows exactly what is happening.

https://serverfault.com/questions/733681/server-sends-ack-in-response-to-syn-causing-a-reset-in-tcp-...

 

So, I am reaching out to the Live forum for help. 

 

Is there a way to generate a reset from the palo alto in this scenerio (an hping3 type command?)

Is there a way to allow that reset from host A through the firewall (maybe turn off state inspection).  Can you turn off state inspection for just one src/dst pair?

 

Thanks


Accepted Solutions
Highlighted
L7 Applicator

What you're referring to is called a Challenge ACK, described in RFC-5961 (Section 4).

 

In a nutshell, it states that if a new SYN is received the server should respond with a new ACK with the most recently sent acknowledgement number as you see. The client will send the reset, which (if it makes it to the server) should free the socket on the server allowing another new SYN to be sent.

 

The RFC is explicit about this (4.2):

   1) If the SYN bit is set, irrespective of the sequence number, TCP
      MUST send an ACK (also referred to as challenge ACK) to the remote
      peer:

      <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>

      After sending the acknowledgment, TCP MUST drop the unacceptable
      segment and stop processing further.

 

Dropping the RST is actually an intended behavior. While it breaks the Challenge ACK mechanism, it prevents out-of-band spoofed addresses from correctly guessing the 4-tupple and killing the connection prematurely.

 

Starting with PAN-OS 8.0.7 (and higher, also included in all versions of 8.1) you would need to enter the following commands:

 

 

> configure
# set deviceconfig setting tcp allow-challenge-ack yes 
# commit

 

That is a global setting, so you can't apply it for just one set of devices. Keep in mind that it also removes the mitigation for a spoofed RST in the current TCP receive window, but that is a pretty tough thing to guess.

View solution in original post


All Replies
Highlighted
L7 Applicator

What you're referring to is called a Challenge ACK, described in RFC-5961 (Section 4).

 

In a nutshell, it states that if a new SYN is received the server should respond with a new ACK with the most recently sent acknowledgement number as you see. The client will send the reset, which (if it makes it to the server) should free the socket on the server allowing another new SYN to be sent.

 

The RFC is explicit about this (4.2):

   1) If the SYN bit is set, irrespective of the sequence number, TCP
      MUST send an ACK (also referred to as challenge ACK) to the remote
      peer:

      <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>

      After sending the acknowledgment, TCP MUST drop the unacceptable
      segment and stop processing further.

 

Dropping the RST is actually an intended behavior. While it breaks the Challenge ACK mechanism, it prevents out-of-band spoofed addresses from correctly guessing the 4-tupple and killing the connection prematurely.

 

Starting with PAN-OS 8.0.7 (and higher, also included in all versions of 8.1) you would need to enter the following commands:

 

 

> configure
# set deviceconfig setting tcp allow-challenge-ack yes 
# commit

 

That is a global setting, so you can't apply it for just one set of devices. Keep in mind that it also removes the mitigation for a spoofed RST in the current TCP receive window, but that is a pretty tough thing to guess.

View solution in original post

Highlighted
L1 Bithead

Thank you, that is exactly what I needed.

 

I would not do this on a permiter firewall, but something internal, the risk - benifit seems reasonable.

 

 

Highlighted
L1 Bithead

One follow up questions:

 

You mentioned:

"That is a global setting, so you can't apply it for just one set of devices. Keep in mind that it also removes the mitigation for a spoofed RST in the current TCP receive window, but that is a pretty tough thing to guess."

 

Can you expand on this just a bit.

 

It is my understanding if the Palo Alto has an established connection in its session table, a spoofed TCP reset -  4 Tuple with a sequence number within the TCP window, and window size equal to or smaller then sessions used,  will make it through the Palo Alto and reset the connection.  Can you confirm this is correct.  

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CllyCAC

 

 

The Challange ack change setting seems like it would allow a reset crafted to make it through the firewall when there is NOT an established  connection in the session table, assuming that 4 Tuple and sequence number/window is in line with the ACK it just saw.  Which seems pretty unlikley.  In summary, I am thinking this feature allows the RESET to make it through during the session start phase of the connection, while the ability to spoof resets on an established connection is inherent risk stated above always exists.

 

 

 

 

 

Highlighted
L7 Applicator

> It is my understanding if the Palo Alto has an established connection in its session table, a spoofed TCP reset -  4 Tuple with a sequence

> number within the TCP window, and window size equal to or smaller then sessions used,  will make it through the Palo Alto and reset the

> connection.  Can you confirm this is correct.  

 

Yes, you are correct. There is no mitigation for that, regardless of the challenge-ack setting I described earlier. Sorry about the confusion.

 

 

> The Challange ack change setting seems like it would allow a reset crafted to make it through the firewall when there is NOT an

> established  connection in the session table, assuming that 4 Tuple and sequence number/window is in line with the ACK it just saw. 

 

This is also correct. You stated it much more plainly (and correctly) than I had :)

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!