TCP fast open and Palo Alto

frigault — Tue, 14 May 2024 09:01:07 GMT

as far as I could test there is no way to make TCP fast open work through a Palo Alto fw (at least, since 9.1 which seemed to work. It tried 10.2 and 11.2 and all my tests fail there).

Whenever a client sends a SYN packet with data, it is transmitted, no matter the zone protection profile, no matter whether the "TCP SYN with Data" option in the zone profile is checked or not.

When the server replies a SYN-ACK acknowledging data, the fw drops the packet (and logs a "TCP reassembly failed: ret 2" in the debug logs).

The only workaround is to set the "strip TCP option > TCP fast open", entirely deactivating TCP fast open.

Has this regression been reported yet?

Re: TCP fast open and Palo Alto

JayGolf — Wed, 15 May 2024 17:24:30 GMT

Hi @frigault ,

I would recommend creating a support ticket and sharing your findings as this has not specifically been addressed in recent updates.

Re: TCP fast open and Palo Alto

frigault — Thu, 16 May 2024 06:23:20 GMT

filed a support case (03048840). With some extra details:

This should be trivial to reproduce but here are some details:
I am reproducing using curl:
curl --tcp-fastopen 10.224.123.2
curl --tcp-fastopen 10.224.123.2

the capture from server side:

07:52:38.824058 IP 192.168.254.1.33362 > 10.224.123.2.http: Flags [S], seq 1997538034:1997538110, win 32120, options [mss 1460,sackOK,TS val 2596799303 ecr 0,nop,wscale 7,tfo cookie feaf99536833786f,nop,nop], length 76: HTTP: GET / HTTP/1.1
<----------- this is a syn with data and a valid tfo cookie. It is transmitted no matter the zone protection profile.
07:52:38.824094 IP 10.224.123.2.http > 192.168.254.1.33362: Flags [S.], seq 3691388850, ack 1997538111, win 31856, options [mss 1460,sackOK,TS val 684199578 ecr 2596799303,nop,wscale 7], length 0
<---------- this is the syn ack from the server. My server does not push data but it acknowledges the data received (ack=1997538111)
07:52:38.824877 IP 10.224.123.2.http > 192.168.254.1.33362: Flags [P.], seq 1:413, ack 1, win 249, options [nop,nop,TS val 684199578 ecr 2596799303], length 412: HTTP: HTTP/1.1 200 OK
07:52:39.870373 IP 10.224.123.2.http > 192.168.254.1.33362: Flags [S.], seq 3691388850, ack 1997538111, win 31856, options [mss 1460,sackOK,TS val 684200624 ecr 2596799303,nop,wscale 7], length 0
<------- server keeps replying and always acknowledging the sent data (ack=1997538111)
07:52:41.918347 IP 10.224.123.2.http > 192.168.254.1.33362: Flags [S.], seq 3691388850, ack 1997538111, win 31856, options [mss 1460,sackOK,TS val 684202672 ecr 2596799303,nop,wscale 7], length 0
^C

from Palo Alto debug logs, the SYN-ACK is dropped because of TCP reassembly
== 2024-05-16 07:51:17.330 +0200 ==
Packet received at fastpath stage, tag 25, type ATOMIC
Packet info: len 74 port 17 interface 17 vsys 1 TC 0
wqe index 17336 packet 0x0xd022401100, HA: 0, IC: 0
Packet decoded dump:
L2: 00:15:5d:34:31:21->00:15:5d:34:31:22, type 0x0800
IP: 10.224.123.2->192.168.254.1, protocol 6
version 4, ihl 5, tos 0x00, len 60,
id 0, frag_off 0x4000, ttl 64, checksum 12278(0xf62f)
TCP: sport 80, dport 33362, seq 3691388850, ack 1997538111,
reserved 0, offset 10, window 31856, checksum 47318,
flags 0x12 ( SYN ACK), urgent data 0, l4 data len 0
TCP option:
00000000: 02 04 05 b4 04 02 08 0a 28 c8 0e 9a 9a c8 03 47 ........ (......G
00000010: 01 03 03 07 ....
Flow fastpath, session 25 s2c (set work 0xd021414d00 exclude_video 0 from sp 0xd03417f480 exclude_video 0)
IP checksum valid
* Dos Profile NULL (NO) Index (0/0) *
2024-05-16 07:51:17.330 +0200 debug: pan_tcp_reass(pan_reass.c:3253): REASS: arg.seqno=3691388850 wqe-id=17336
2024-05-16 07:51:17.330 +0200 debug: pan_tcp_reass(pan_reass.c:3265): REASS: work 0xd021414d00 session=25 seqno=3691388850 ackno=1997538111 tcplen=0 l4plen=0 flags 0x12 s2c state 0
2024-05-16 07:51:17.330 +0200 debug: pan_tcp_reass(pan_reass.c:3803):
session=25 work 0xd021414d00 payload len 0, tcp data len 0, ret 2
TCP reassembly failed: ret 2

Another reproducer, this time using a publicly available server supporting TCP fast open
$ sysctl net.ipv4.tcp_fastopen net.ipv4.tcp_fastopen_blackhole_timeout_sec
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 0
$ curl --tcp-fastopen ocsp.usertrust.com
^C
$ curl --tcp-fastopen ocsp.usertrust.com
^C
$ curl --tcp-fastopen ocsp.usertrust.com
^C

Re: TCP fast open and Palo Alto

frigault — Thu, 27 Mar 2025 18:46:13 GMT

https://docs.paloaltonetworks.com/pan-os/11-2/pan-os-release-notes/pan-os-11-2-5-known-and-addressed-issues/pan-os-11-2-5-addressed-issues

mentions

PAN-258149	Fixed an issue where the firewall dropped the SYN-ACK when using the TCP Fast Open option.

from my tests it seems the issue is fixed in this version.

topic Re: TCP fast open and Palo Alto in General Topics

TCP fast open and Palo Alto

Re: TCP fast open and Palo Alto

Re: TCP fast open and Palo Alto

Re: TCP fast open and Palo Alto