topic Aged-out traffic accessing a common web in Next-Generation Firewall Discussions

Aged-out traffic accessing a common web

JoseCortijo — Mon, 10 Oct 2022 14:16:46 GMT

Hi,

recently I am facing an aged-out case for a typical web site, reachable without any issue from 4G for example.

the traffic is not decrypted and after reading many articles I am running out of ideas.

Checking the session info I saw a mismatch between the sport in the c2s flow and the dport in the s2c flows. Should not be the same port number? could that be a reason for the incomplete error?

any idea on how could I troubleshoot this issue?

many thanks

show session id 121955 Session 121955 c2s flow: source: 192.168.20.208 [Internal] dst: 217.22.97.1 proto: 6 sport: 62715 dport: 443 state: INIT type: FLOW src user: DOMAIN\user dst user: unknown s2c flow: source: 217.22.97.1 [External] dst: X.Y.Z.Z (my public IP) proto: 6 sport: 443 dport: 44353 state: INIT type: FLOW src user: unknown dst user: DOMAIN\user start time : Mon Oct 10 15:31:34 2022 timeout : 5 sec total byte count(c2s) : 198 total byte count(s2c) : 0 layer7 packet count(c2s) : 3 layer7 packet count(s2c) : 0 vsys : vsys1 application : incomplete rule : Unfiltered Internet service timeout override(index) : False session to be logged at end : True session in session ager : False session updated by HA peer : False address/port translation : source nat-rule : NAT-OUT-1(vsys1) layer7 processing : enabled URL filtering enabled : True URL category : any session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ae1 egress interface : ae2.6 session QoS rule : N/A (class 4) tracker stage firewall : Aged out end-reason : aged-out

Re: Aged-out traffic accessing a common web

aleksandar.astardzhiev — Mon, 10 Oct 2022 14:51:52 GMT

Hi @JoseCortijo ,

Let break down the output:

- Aged out means that firewall have removed this connection from its connection table because the relevant timer for this session expired. For UDP traffic it is normal to see aged-out, because the protocol is stateless and firewall cannot identify when session is actually gracefully closed.

- From the output we can see it is TCP (protocol 6), and the timeout was after 5 seconds, which is matching the default "TCP init" timer ("Maximum length of time, in seconds, between receiving the SYN and SYN-ACK before starting the TCP handshake timer")

- From the output it seem source NAT is being performed - nat rule is populated with the name of the matched NAT rule, as well as translated address is used for the expected return traffic (s2c flow)

- However it looks like no return traffic is reaching the firewall - total bytes count (s2c) and layer7 packet count (s2c) are both 0. This is clear indication that retunr traffic is either not reaching the firewall (most probably) of for some other reason FW is not able to match the returning traffic correctly to this session.

I would suggest you to first check and confirm that the public IP address you are using for that NAT rule is corrrect

- It is correctly routed back to your firewall if you are using additional NAT range from your ISP

- It is not used by other device in the network which could cause IP conflict

- If the destination server is using some kind of IP whitelisting if configured to allow your IP address. You mentioned it is "common web server" I would expect that is something that is open for any public addresses

P.S. It is normal to see different sport in c2s from dport in s2c. That is because your NAT rule is probably configured to use "dynamic IP and port" type of source NAT. In this case firewall is using the source port as matching criteria to identify for which session the return traffic belongs to. If FW was not translating the source port and forwarded the original source port, it is very possible that two internal hosts generate traffic with same source port, so firewall will not be able to identify to which host it should forward the reply.

Re: Aged-out traffic accessing a common web

JoseCortijo — Tue, 11 Oct 2022 09:05:42 GMT

Hi @aleksandar.astardzhiev ,

thanks for your quick reply and long explanation, very instructive.

there are no more additional NAT from our ISP that I am aware of and the IP is not used by any other device in the network.

In fact the public IP shown is used by our whole office to access the Internet and this site is the only one having connectivity issues. The target host is a public website of an electricity firm, reachable from my phone without issues.

I did some packecapture from the FW and indeed I only see packets leaving paloalto and attempts of retransmission. Running out of ideas...

I will ask the local provider to confirm there is no filtering from its side.

cheers

Re: Aged-out traffic accessing a common web

aleksandar.astardzhiev — Wed, 12 Oct 2022 12:05:11 GMT

Hey @JoseCortijo ,

In my experience it is more common the destination server to perform some kind of filtering/allowlisting, rather for the ISP to block you.(In general ISP don't have any benefit to restrict your traffic, while at the same time will want to keep you as customer and provide ways to connecting to all parts of the world).

I am wondering if wrong IP geo-location information is not the problem (if the electricity company allow access only from given region/country). If you check your public NAT IP in sites like https://www.iplocation.net/ do you see correct geo-location information? No need to be exact, at least to show the correct country?

Re: Aged-out traffic accessing a common web

JoseCortijo — Thu, 13 Oct 2022 08:03:33 GMT

Hi @aleksandar.astardzhiev ,

the ISP did a connection test and confirmed that it is our public IP that is blocked at the server level. I wonder what might be the reason behind it. I checked our public IP on the site you mentioned and it shows Spain.

My issue now is how to reach the technicians behind the domain. in whois everything is hidden and only some telephone numbers are offered. I am imagining a non-technical receptionist picking up the phone and myself trying to explain to her all this issue. 😕

Thanks a lot for your support.

Re: Aged-out traffic accessing a common web

jacobcavaness — Tue, 21 May 2024 16:19:39 GMT

Wanted to leave this comment here in case someone else encounters this issue.

We had a similar problem where some websites would load while others wouldn't. We could ping the website in question, but the webpage itself would never load, displaying "aged-out or tcp-fin." Initially, we thought it was a routing issue. It turned out to be an MTU issue on an interface.