SYN ACK RST Loop on inside and SYN ACK only - source port re-use - hung session

[william.dolbow]

I have had this happen twice so far, I am wondering how others have solved for it.

Host A -> Palo Alto -> Host B

Host A, cycles through its source ports frequently - every couple of minutes.

Host B has long  or no TCP keep alive timers.

Host A:src port X connects to Host B on port 443.

Host A has an issue, reboots, lost power, app crashes, etc.  But it does NOT close the session.  Host B shows the connection in the netstat.  

Host A:src port X tries to connect to host B on port 443.  Host B responds with an ACK, this is per RFC.  That ACK is passed back to host A, Host A responds to the ACK with a RST.  The firewall drops the reset, and the connection is stuck.  Every time the host cycles through that source port it never clears.

Looking on the firewall on the A side, all I see is

Syn ->

<- ACK

RST->

For like 6 tries from the host on this source port, then is moves on until it gets back around to this port and it does it again and again.  No way to clear it.

On the B side, I see this

Syn ->

<- ACK

No reset, the firewall is eating the reset.

I tried putting a deny rule in with reset in both directions, but it only resets on the A side.

Clearing the session in the session table does not work.

I have tried to spoof reset packets and they are not getting through the firewall.

The only thing I have been able to do is have the server guy either clear the session or reboot it if they dont have the capabliity to clear a stale session.  Ask them to turn on keepalives to try to prevent, but this does not seem right.  

This site shows exactly what is happening.

https://serverfault.com/questions/733681/server-sends-ack-in-response-to-syn-causing-a-reset-in-tcp-...

So, I am reaching out to the Live forum for help. 

Is there a way to generate a reset from the palo alto in this scenerio (an hping3 type command?)

Is there a way to allow that reset from host A through the firewall (maybe turn off state inspection).  Can you turn off state inspection for just one src/dst pair?

Thanks

[gwesson · Accepted solution]

What you're referring to is called a Challenge ACK, described in RFC-5961 (Section 4).

In a nutshell, it states that if a new SYN is received the server should respond with a new ACK with the most recently sent acknowledgement number as you see. The client will send the reset, which (if it makes it to the server) should free the socket on the server allowing another new SYN to be sent.

The RFC is explicit about this (4.2):

   1) If the SYN bit is set, irrespective of the sequence number, TCP
      MUST send an ACK (also referred to as challenge ACK) to the remote
      peer:

      <SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>

      After sending the acknowledgment, TCP MUST drop the unacceptable
      segment and stop processing further.

Dropping the RST is actually an intended behavior. While it breaks the Challenge ACK mechanism, it prevents out-of-band spoofed addresses from correctly guessing the 4-tupple and killing the connection prematurely.

Starting with PAN-OS 8.0.7 (and higher, also included in all versions of 8.1) you would need to enter the following commands:

> configure
# set deviceconfig setting tcp allow-challenge-ack yes 
# commit

That is a global setting, so you can't apply it for just one set of devices. Keep in mind that it also removes the mitigation for a spoofed RST in the current TCP receive window, but that is a pretty tough thing to guess.

[william.dolbow]

Thank you, that is exactly what I needed.

I would not do this on a permiter firewall, but something internal, the risk - benifit seems reasonable.

[william.dolbow]

One follow up questions:

You mentioned:

"That is a global setting, so you can't apply it for just one set of devices. Keep in mind that it also removes the mitigation for a spoofed RST in the current TCP receive window, but that is a pretty tough thing to guess."

Can you expand on this just a bit.

It is my understanding if the Palo Alto has an established connection in its session table, a spoofed TCP reset -  4 Tuple with a sequence number within the TCP window, and window size equal to or smaller then sessions used,  will make it through the Palo Alto and reset the connection.  Can you confirm this is correct.  

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000CllyCAC

The Challange ack change setting seems like it would allow a reset crafted to make it through the firewall when there is NOT an established  connection in the session table, assuming that 4 Tuple and sequence number/window is in line with the ACK it just saw.  Which seems pretty unlikley.  In summary, I am thinking this feature allows the RESET to make it through during the session start phase of the connection, while the ability to spoof resets on an established connection is inherent risk stated above always exists.

[gwesson]

> It is my understanding if the Palo Alto has an established connection in its session table, a spoofed TCP reset -  4 Tuple with a sequence

> number within the TCP window, and window size equal to or smaller then sessions used,  will make it through the Palo Alto and reset the

> connection.  Can you confirm this is correct.  

Yes, you are correct. There is no mitigation for that, regardless of the challenge-ack setting I described earlier. Sorry about the confusion.

> The Challange ack change setting seems like it would allow a reset crafted to make it through the firewall when there is NOT an

> established  connection in the session table, assuming that 4 Tuple and sequence number/window is in line with the ACK it just saw. 

This is also correct. You stated it much more plainly (and correctly) than I had 🙂

[Sec101]

This post is gold.

[dtran]

I run into this exact same issue a few weeks ago.  There is a very good read on this:  https://www.networkdefenseblog.com/post/wireshark-tcp-challenge-ack