NFS times out after 60 minutes

[ddaniels]

NFS times out after 60 minutes even when setting allow all ports between servers. Before that the NFS protocol got it's session timeout increased to a few days, but it would still timeout after 60 minutes. iptables did not have this problem with the same two servers and nothing has changed other than replacing iptables with a PAN.

[panos]

what is your version of OS ?

[rmonvon]

Hi...Maybe the application timeouts need to be adjusted on the PA device.  You can go under Objects ==> Applications, search for nfs and review its timeouts.  Here's the nfs timeouts on my PA and they can be increased by selecting the 'Customize...'.

Thanks.

[mikand]

However that timeout is due to inactivity... if the NFS rule times out even with packets flowing then it sounds like some other issue.

[igor_mamuzic]

I had a similar issue with NFS trough PA. Problem was that periodically NFS clients could not communicate with the NFS share on the server after they successfully mounted a disk. After analyzing packet capture we noticed that NFS clients are reusing the same source port while initiating connection with the server while previous connection is still in "FIN handshake" (TCP FIN handshake still in progress) state. Firewall started to count its FIN timeout for the session and during that timeout period the client sent a new SYN packet using the same source and destination ports and addresses. Time gap between initiating FIN handshake of old connection and TCP SYN handshake for the new connection was less then 1 second. We think that these sub-second connection attempts was due to client needed to maintain a permanent TCP connection with NFS server, but something was obviously clearing this connection periodically (maybe it was a server). Anyway,  we could'nt do anything about it on the firewall because minimum value for FIN timeout on PA firewall is 1 second. So, the firewall was drooping new TCP SYN packets, because it had the same session still in the session table. We noticed that by analyzing packet capture from the PA firewall at "drop" stage.

The solution was to change the NFS mount mode into "automount" which worked in a way that for each new mount attempt NFS creates a new connection. Since the client in our case had to send some data to NFS share for every 15 minutes, net effect of switching to automount was that we needed a new TCP connection only when we needed to access the NFS share. After the client is done using the share, TCP connection has been terminated by proper TCP FIN sequence.

Check Point has a feature to deal with situations like this and it's called TCP Smart Connection Reuse. Interesting feature. I learned that because NFS traffic was migrated from Check Point to PA and you can imagine how we as PA partners felt when CP guys said "I told you that this PA stinks":) :smileysilly:

Anyway, monitor your NFS session from CLI (with 'show session' command) constantly until it breaks to see if timeout switched to 30 sec (if defaults are still on). This could mean that FIN timeout has been activated. Also, collect a packet capture that contains at least two contiguous NFS session breaks (failures) and look for reused source ports.

Igor

[mikand]

Sounds more like a not so smart reuse 🙂

Next time you run into these CP guys you can ask them on how well their userid function works (or for that matter how many mbit/s you have left in throughput when you enable all IPS signatures 😉

Are there any differents in the resuse of ports for NFS regarding TCP vs UDP? Or for that matter hardmount vs softmount and such?

[igor_mamuzic]

Yeah, well, you know, how should I put this:) It's pretty smart reuse:) because NFS worked like a charm before it was moved to PA. So, I agree that maybe this NFS setup with port reuse is not something that should be encouraged. On the other side, if something works all right on the old system (firewall) and new firewall breaks it without any possibility to fix this on the firewall itself then I don't see any point talking about how other features are better comparing with old firewall. It's like you have an old, rusty 4wd which serves you very vell, then you replace it with brand new SUV which cannot take you to your county side home because it lacks true off road capabilities and then the reseller says, yes, but look it has auto A/C, GPS Nav,etc. and that's why it's so much better then the one you had before

Regarding your quesrtion about TCP vs UDP, we didn't tried switching the conversation to UDP, because we managed to make it works with automount and it was about "don't touch without good reason" mission critical environment, so we didn't played with it any further.

Igor

[Retired Member]

I am not familiar with CP port reuse option, but am curious if it basically disables TCP window or sequence checking. On PANW firewalls you can disable TCP window checking to work around port reuse situations. One major issue with hosts that re-use source ports for connections is that second session would use new Seq/Ack values which would likely fall outside of our normal TCP window checks. On other vendor platforms, this may also be referred to as TCP sequence checking. I normally recommend to correct the behavior to not re-use source ports at the source since this is inherently insecure behavior. But if that is not possible, then disabling the TCP window checking functionality in PAN-OS may be the best alternative.

There are two ways to disable the TCP window checking.

1. Globally with configuration:
# set deviceconfig setting tcp asymmetric-path bypass

# commit

2. Per zone within Zone Protection Profile > Packet-Based Attack Prevention.
Set Asymmetric Path to "bypass". Then apply to the zone in which client originates the session followed by commit.

The risk in disabling TCP Window checking is that you can potentially open up the firewall to man-in-the-middle attacks. This is why I recommend to correct the problem at the source.

-Richard