Forward segments exceeding TCP content inspection queue

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Forward segments exceeding TCP content inspection queue

L7 Applicator

Hi,

 

On a new PA-3020 Firewallcluster I decided to disable the default setting "Forward segments exceeding TCP content inspection queue". Practically everything was working as it should. But onfortunately the devil is in the details. I had very few connections, specially http downloads, which where causing problems. Sometimes the same download was working, sometimes it was just somewhere between slow and really slow and sometimes the download was stopping completely.

 

The following is written in PaloAlto Best Practices for securing your network from layer 4 and layer 7 evasions:

"By default, when the TCP or UDP content inspection queue is full, the firewall skips Content-ID inspection for TCP segments or UDP datagrams that exceed the queue limit of 64. By disabling these options, the firewall instead drops TCP segments and UDP datagrams when the corresponding TCP or UDP content inspection queue is full.
Disabling these options can result in performance degradation and some applications may incur loss of functionality, particularly in high-volume traffic situations."
 
So because of these Problems I was forced to turn on this setting again. But now I am not really sure what exact risk does this mean or in which cases is enabling this setting effectively going to be a security issue?
And there is also this question: Shouldn't a PA-3020 be able to process an 100 Mbit/s download with this setting turned off? (at the time when a download failed the active sessions were at about 5000 while 3500 of them were decrypted)
 
I would appreciate your opinions and inputs to this.
 
Regards,
Remo
 
PS: The same questions also apply to the setting for UDP
7 REPLIES 7

Cyber Elite
Cyber Elite

The queue is used to enable ctd to scan across fragmentation, missing or out of order segments. If there are high amounts of these in a session, the queue for that session might get exceeded and the configured action will be taken to clear the queue.

 

if this happens a lot on valid sessions, it might be good to investigate the cause and try to fix that (by for example enabling TCP MSS and lowering the MTU)

 

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

@reaper is this max queue of 64 per session?

Enterprise Architect, Security @ Cloud Carib Ltd
Palo Alto Networks certified from 2011

yes, each session has an individual queue; so one application may be impacted while another is not, depending on the circumstances

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

Is there cli command to get current queue length for different sessions.

Let's say top 10 sessions with biggest queue?

Enterprise Architect, Security @ Cloud Carib Ltd
Palo Alto Networks certified from 2011

you can check the overall state of CTD

> debug dataplane show ctd memory-state 

not sure if you can go as far as to check per session as that's gonna put you in a highly volatile environment

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

All right so I will now do a deep dive into MTU/MSS troubleshooting. It remains a little strange to me (probably because of not enough knowledge about MTU/MSS), but for 1.5 month there where absolutely no complaints from the customer about connection problems. This one download, actually one website where different downloads where provided, was the only problem.
Even this troubleshooting took a while because I wasn't thinking at all that it could be related to this (also because everything else was working).
Because of no threat logs and no other blocked connections, I did the next test with disabling various settings in the zone protection profile up to disable the zone protection completely. Without success. So the next step was a "flow basic" debugging where I have seen in the counters that there where ctd_exceed_quque drops. Then the situation was pretty clear why the download was failing.

So thanks @reaper for pointing me to the right direction for the next steps in the troubleshooting process

Global counters were actually my bread and butter during most troubleshooting sessions
They're easy to obtain (set filters, run global counters with delta) and give you immediate feedback on what's happening with your session

I'd recommend using them more often 🙂
Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization
  • 10212 Views
  • 7 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!