TCP TimeOut caused by the PA?

[jhickey]

We have a video app that is streaming through our Palo Alto firewall on port 80. Everyone once in a while the session fails and can only be revived by hitting refresh in the browser. I am dealing with a network manager that's convinced the PAs are Resetting the session.

Before I go through the hassle of creating override policies for port 80 with higher timeouts I'd like to know if there is anything in the logs that would show that the PALOALTO Terminated the connection. The CISCO Firewalls put something in their logs (TCP Timeout). I'm hoping something similar exists.

Thanks,

Justin

[rmonvon]

Hi...You can review the traffic logs, search for the video session that ended, expand the log details and veirfy the start & stop time of the session and note the elapse time.  Once you have verified the session, note the application name.  Then navigate to Objects ==> Applications, look up the application and check its TCP timeout.  If the TCP timeout is close to the elapse time, then it is likely the application was terminated as a result of the TCP timeout for the app.   You can then modify & extend the default timeout for the app.  Thanks.

[mikand]

You can also do the same if you tap the traffic (using span-port) before and after the PAN and then check the generated PCAP in wireshark or tcpdump.

[jhickey]

Hi, Thanks for the responses. As I search the forum I realized that I've posted the same question 3 times over the past year.  I know both of your suggestions will work, it just seems over labor intensive to me to do this every time we suspect a timeout from the PAs. How hard would it be for the PA Developers to add a log condition for when the connection is timed out by the firewall ?

Thanks,
Justin

[mikand]

log on session end (which is the default if you enable logging) should bring you this.

The problem here is to find out why the session ended and in my opinion using pcap will bring you the proofs you need of whats actually going on in the wire.

[jhickey]

I do log on session end. There is nothing that states TCP TIMEOUT like in the ASA.

I do capture PCAPs but even there its tough to see the RST coming from the firewall. It's a needle in a haystack and I may be asked to do this a few times a quarter. When the firewall is creating an outage (intentionally) I think it should log it . No?

[rmonvon]

To preserve resource many applications will not end their sessions and the firewall has to end them via timeouts.  The most common app is web browsing where the browser typically will not waste resource to send FIN or RST to end their sessions.  Timeouts conducted by the firewall is quite common.

For video apps, if the control session is not transmitting regularly (keepalive) then the firewall may timeout the session thinking that it is an idle connection.  I recommend you test by launching a video session, monitor what apps are detected, and extended the timeouts for those apps.

Thanks.

[garryshtern]

The issue here is that a firewall will silently drop the session from the table w/o notifying either server or client.  Thus, if a client that opens long-lived connection via the FW and FW tears down the connection, next time client or server attempt to re-use it they'll be restransmitting for several seconds or longer (depending on the retransmit settings).

Is there a way to enable something on the firewall so it sends RST if/when it removes the session from a table.  Or, at the very least, send RST next time either party tries to communicate on the removed session...

[sdurga]

Garry,

I do not think paloalto will send any RST packets when we discard a session due to timeout or due to security policy deny. That is my understanding at least from this thread.

[garryshtern]

Yeah, it looks like it's a oversight...

[mikand]

Regarding your initial problem the timeout in PA should not occur as long as there are traffic being sent in either direction.

So a question here might be why this streaming app obviously stops sending packets on its own so a timeout occurs unless there is something else malfunctioning on the road?

Edit: Or is there some kind of "max life" timeout aswell (to deal with slowloris etc)?

[jhickey]

Im not questioning the need for the firewall to timeout connections. It's a stateful box and cleaning up old sessions is a requirement. What often happens is firewalls dont understand a certain type of control traffic and enforce a timeout that is too short, terminating an active connection and causing the app to error out. That's going to happen on any firewall.

We as technicians need clear indications as to why a session was terminated. Just about every firewall out there will put spit out a TCP Timeout entry into some kind of log when this happens. If the log entry is there we will instantly know to create a custom application class with a longer timeout or with no timeout. Without the log entry we're left with more troubleshooting and time lost testing other theories.

I hope they fix this because its kind of a big deal. Also if they could fix reporting and add a good log viewer while they are at it that would be great. (in my best Lundberg voice)

[Black_Banes]

Hello All, 

We have a similar issue to the one described on this thread where two endpoints (Client and Server) are trying to resume from a long period of inactivity but since the firewall had silently dropped the aged out session, the communication fails between the hosts and we have to manually restart the application. So the issue is, the firewall silently tears down the session once the expiration time is reached, without any whatsoever notification. As a result, the two endpoints attempt to resume from the last session, assuming the session is still active but the firewall discards the packets expecting to receive a new SYN message marking the start of a new session. My question is: Does PALO ALTO has any workaround other than increasing the Default session Timeout to a Max Value of 7Days?  Can the Palo acts as a proxy and offload the TCP session?

Thanks

[ddelcourt]

This is an age old discussion around application behavior... a couple things to consider, not all applications behave the same and I'm sure we all have war stories of sending a TCP-RST caused an application to blow up 🙂 so, no perfect answers for everything which is why if there is no traffic seen during the defined session timeout period it will be aged out and future traffic not starting with the appropriate TCP handshake will be dropped.

If the FW terminates a session then the reason it does so should be captured in the log at session end as described in this thread, see https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslo... for a run down (PANOS 8.1) of how the log is broken down. It is a little cryptic, but there is a parameter tracked for "aged out" of a session as a "reason" for session end.

As to what can I do? The first thing to check is to see if there is an App-ID for the application, Application Research Center would be the place to start.

If not, then you could create a custom App-ID for that specific flow and manage the timeout and other characteristics.

If all of that is too much to do right now then depending on your version of PANOS you can create a custom Service Object and manage the timeout value, https://docs.paloaltonetworks.com/pan-os/8-1/pan-os-web-interface-help/objects/objects-services.html. This Service object could then be used in the specific security rule for that source(s)/destination(s) requiring the different timeout and/or other characteristics.

In general, changing the default UDP/TCP timeout is not a good idea as that becomes pervasive across everything which is probably not desired nor is it generally good security hygiene... hth

[NisterioHD]

Thanks, @ddelcourt for the great input. I am wondering if there is a way to have the Palo Alto send a RST/FIN message to the endpoints to notify them when the session age-out instead of just dropping the session silently?!  I know you mentioned the alternative of creating a  custom App-ID/ Service object but if that requires to modify the normal behavior of the firewall then it is not an option in my case. Let me know, Cheers.