topic Re: Forward segments exceeding TCP content inspection queue in General Topics

Forward segments exceeding TCP content inspection queue

Remo — Wed, 24 May 2017 08:57:25 GMT

Hi,

On a new PA-3020 Firewallcluster I decided to disable the default setting "Forward segments exceeding TCP content inspection queue". Practically everything was working as it should. But onfortunately the devil is in the details. I had very few connections, specially http downloads, which where causing problems. Sometimes the same download was working, sometimes it was just somewhere between slow and really slow and sometimes the download was stopping completely.

The following is written in PaloAlto Best Practices for securing your network from layer 4 and layer 7 evasions:

"By default, when the TCP or UDP content inspection queue is full, the firewall skips Content-ID inspection for TCP segments or UDP datagrams that exceed the queue limit of 64. By disabling these options, the firewall instead drops TCP segments and UDP datagrams when the corresponding TCP or UDP content inspection queue is full.

Disabling these options can result in performance degradation and some applications may incur loss of functionality, particularly in high-volume traffic situations."

( https://www.paloaltonetworks.com/documentation/80/pan-os/pan-os/threat-prevention/best-practices-for-securing-your-network-from-layer-4-and-layer-7-evasions )

So because of these Problems I was forced to turn on this setting again. But now I am not really sure what exact risk does this mean or in which cases is enabling this setting effectively going to be a security issue?

And there is also this question: Shouldn't a PA-3020 be able to process an 100 Mbit/s download with this setting turned off? (at the time when a download failed the active sessions were at about 5000 while 3500 of them were decrypted)

I would appreciate your opinions and inputs to this.

Regards,

Remo

PS: The same questions also apply to the setting for UDP

Re: Forward segments exceeding TCP content inspection queue

reaper — Wed, 24 May 2017 10:00:14 GMT

The queue is used to enable ctd to scan across fragmentation, missing or out of order segments. If there are high amounts of these in a session, the queue for that session might get exceeded and the configured action will be taken to clear the queue.

if this happens a lot on valid sessions, it might be good to investigate the cause and try to fix that (by for example enabling TCP MSS and lowering the MTU)

Re: Forward segments exceeding TCP content inspection queue

Raido_Rattameister — Wed, 24 May 2017 12:34:24 GMT

@reaper is this max queue of 64 per session?

Re: Forward segments exceeding TCP content inspection queue

reaper — Wed, 24 May 2017 12:42:27 GMT

yes, each session has an individual queue; so one application may be impacted while another is not, depending on the circumstances

Re: Forward segments exceeding TCP content inspection queue

Raido_Rattameister — Wed, 24 May 2017 14:15:55 GMT

Is there cli command to get current queue length for different sessions.

Let's say top 10 sessions with biggest queue?

Re: Forward segments exceeding TCP content inspection queue

reaper — Wed, 24 May 2017 14:21:23 GMT

you can check the overall state of CTD

> debug dataplane show ctd memory-state

not sure if you can go as far as to check per session as that's gonna put you in a highly volatile environment

Re: Forward segments exceeding TCP content inspection queue

Remo — Wed, 24 May 2017 18:46:06 GMT

All right so I will now do a deep dive into MTU/MSS troubleshooting. It remains a little strange to me (probably because of not enough knowledge about MTU/MSS), but for 1.5 month there where absolutely no complaints from the customer about connection problems. This one download, actually one website where different downloads where provided, was the only problem.
Even this troubleshooting took a while because I wasn't thinking at all that it could be related to this (also because everything else was working).
Because of no threat logs and no other blocked connections, I did the next test with disabling various settings in the zone protection profile up to disable the zone protection completely. Without success. So the next step was a "flow basic" debugging where I have seen in the counters that there where ctd_exceed_quque drops. Then the situation was pretty clear why the download was failing.

So thanks @reaper for pointing me to the right direction for the next steps in the troubleshooting process

Re: Forward segments exceeding TCP content inspection queue

reaper — Wed, 24 May 2017 18:46:13 GMT

Global counters were actually my bread and butter during most troubleshooting sessions
They're easy to obtain (set filters, run global counters with delta) and give you immediate feedback on what's happening with your session

I'd recommend using them more often 🙂