Aged-out traffic accessing a common web

[JoseCortijo]

Hi,

recently I am facing an aged-out case for a typical web site, reachable without any issue from 4G for example.

the traffic is not decrypted and after reading many articles I am running out of ideas. 

Checking the session info I saw a mismatch between the sport in the c2s flow and the dport in the s2c flows. Should not be the same port number? could that be a reason for the incomplete error?

any idea on how could I troubleshoot this issue?

many thanks

show session id 121955
Session          121955

        c2s flow:
                source:      192.168.20.208 [Internal]
                dst:         217.22.97.1
                proto:       6
                sport:       62715           dport:      443
                state:       INIT            type:       FLOW
                src user:    DOMAIN\user
                dst user:    unknown

        s2c flow:
                source:      217.22.97.1 [External]
                dst:         X.Y.Z.Z (my public IP)
                proto:       6
                sport:       443             dport:      44353
                state:       INIT            type:       FLOW
                src user:    unknown
                dst user:    DOMAIN\user

        start time                           : Mon Oct 10 15:31:34 2022
        timeout                              : 5 sec
        total byte count(c2s)                : 198
        total byte count(s2c)                : 0
        layer7 packet count(c2s)             : 3
        layer7 packet count(s2c)             : 0
        vsys                                 : vsys1
        application                          : incomplete
        rule                                 : Unfiltered Internet
        service timeout override(index)      : False
        session to be logged at end          : True
        session in session ager              : False
        session updated by HA peer           : False
        address/port translation             : source
        nat-rule                             : NAT-OUT-1(vsys1)
        layer7 processing                    : enabled
        URL filtering enabled                : True
        URL category                         : any
        session via syn-cookies              : False
        session terminated on host           : False
        session traverses tunnel             : False
        session terminate tunnel             : False
        captive portal session               : False
        ingress interface                    : ae1
        egress interface                     : ae2.6
        session QoS rule                     : N/A (class 4)
        tracker stage firewall               : Aged out
        end-reason                           : aged-out

[aleksandar.astardzhiev]

Hi @JoseCortijo ,

 Let break down the output:

- Aged out means that firewall have removed this connection from its connection table because the relevant timer for this session expired. For UDP traffic it is normal to see aged-out, because the protocol is stateless and firewall cannot identify when session is actually gracefully closed.

- From the output we can see it is TCP (protocol 6), and the timeout was after 5 seconds, which is matching the default "TCP init" timer ("Maximum length of time, in seconds, between receiving the SYN and SYN-ACK before starting the TCP handshake timer")

- From the output it seem source NAT is being performed - nat rule is populated with the name of the matched NAT rule, as well as translated address is used for the expected return traffic (s2c flow)

- However it looks like no return traffic is reaching the firewall - total bytes count (s2c) and layer7 packet count (s2c) are both 0. This is clear indication that retunr traffic is either not reaching the firewall (most probably) of for some other reason FW is not able to match the returning traffic correctly to this session.

I would suggest you to first check and confirm that the public IP address you are using for that NAT rule is corrrect

- It is correctly routed back to your firewall if you are using additional NAT range from your ISP

- It is not used by other device in the network which could cause IP conflict

- If the destination server is using some kind of IP whitelisting if configured to allow your IP address. You mentioned it is "common web server" I would expect that is something that is open for any public addresses

P.S. It is normal to see different sport in c2s from dport in s2c. That is because your NAT rule is probably configured to use "dynamic IP and port" type of source NAT.  In this case firewall is using the source port as matching criteria to identify for which session the return traffic belongs to. If FW was not translating the source port and forwarded the original source port, it is very possible that two internal hosts generate traffic with same source port, so firewall will not be able to identify to which host it should forward the reply.

[JoseCortijo]

Hi @aleksandar.astardzhiev ,

thanks for your quick reply and long explanation, very instructive.

there are no more additional NAT from our ISP that I am aware of and the IP is not used by any other device in the network.

In fact the public IP shown is used by our whole office to access the Internet and this site is the only one having connectivity issues. The target host is a public website of an electricity firm, reachable from my phone without issues.

I did some packecapture from the FW and indeed I only see packets leaving paloalto and attempts of retransmission. Running out of ideas...

I will ask the local provider to confirm there is no filtering from its side.

cheers

[aleksandar.astardzhiev]

Hey @JoseCortijo ,

In my experience it is more common the destination server to perform some kind of filtering/allowlisting, rather for the ISP to block you.(In general ISP don't have any benefit to restrict your traffic, while at the same time will want to keep you as customer and provide ways to connecting to all parts of the world).

I am wondering if wrong IP geo-location information is not the problem (if the electricity company allow access only from given region/country). If you check your public NAT IP in sites like https://www.iplocation.net/ do you see correct geo-location information? No need to be exact, at least to show the correct country?

[JoseCortijo]

Hi @aleksandar.astardzhiev ,

the ISP did a connection test and confirmed that it is our public IP that is blocked at the server level. I wonder what might be the reason behind it. I checked our public IP on the site you mentioned and it shows Spain.

My issue now is how to reach the technicians behind the domain. in whois everything is hidden and only some telephone numbers are offered. I am imagining a non-technical receptionist picking up the phone and myself trying to explain to her all this issue. 😕

Thanks a lot for your support.

[jacobcavaness]

Wanted to leave this comment here in case someone else encounters this issue.

We had a similar problem where some websites would load while others wouldn't. We could ping the website in question, but the webpage itself would never load, displaying "aged-out or tcp-fin." Initially, we thought it was a routing issue. It turned out to be an MTU issue on an interface.