10-10-2022 07:16 AM
recently I am facing an aged-out case for a typical web site, reachable without any issue from 4G for example.
the traffic is not decrypted and after reading many articles I am running out of ideas.
Checking the session info I saw a mismatch between the sport in the c2s flow and the dport in the s2c flows. Should not be the same port number? could that be a reason for the incomplete error?
any idea on how could I troubleshoot this issue?
show session id 121955 Session 121955 c2s flow: source: 192.168.20.208 [Internal] dst: 220.127.116.11 proto: 6 sport: 62715 dport: 443 state: INIT type: FLOW src user: DOMAIN\user dst user: unknown s2c flow: source: 18.104.22.168 [External] dst: X.Y.Z.Z (my public IP) proto: 6 sport: 443 dport: 44353 state: INIT type: FLOW src user: unknown dst user: DOMAIN\user start time : Mon Oct 10 15:31:34 2022 timeout : 5 sec total byte count(c2s) : 198 total byte count(s2c) : 0 layer7 packet count(c2s) : 3 layer7 packet count(s2c) : 0 vsys : vsys1 application : incomplete rule : Unfiltered Internet service timeout override(index) : False session to be logged at end : True session in session ager : False session updated by HA peer : False address/port translation : source nat-rule : NAT-OUT-1(vsys1) layer7 processing : enabled URL filtering enabled : True URL category : any session via syn-cookies : False session terminated on host : False session traverses tunnel : False session terminate tunnel : False captive portal session : False ingress interface : ae1 egress interface : ae2.6 session QoS rule : N/A (class 4) tracker stage firewall : Aged out end-reason : aged-out
10-10-2022 07:51 AM
Hi @JoseCortijo ,
Let break down the output:
- Aged out means that firewall have removed this connection from its connection table because the relevant timer for this session expired. For UDP traffic it is normal to see aged-out, because the protocol is stateless and firewall cannot identify when session is actually gracefully closed.
- From the output we can see it is TCP (protocol 6), and the timeout was after 5 seconds, which is matching the default "TCP init" timer ("Maximum length of time, in seconds, between receiving the SYN and SYN-ACK before starting the TCP handshake timer")
- From the output it seem source NAT is being performed - nat rule is populated with the name of the matched NAT rule, as well as translated address is used for the expected return traffic (s2c flow)
- However it looks like no return traffic is reaching the firewall - total bytes count (s2c) and layer7 packet count (s2c) are both 0. This is clear indication that retunr traffic is either not reaching the firewall (most probably) of for some other reason FW is not able to match the returning traffic correctly to this session.
I would suggest you to first check and confirm that the public IP address you are using for that NAT rule is corrrect
- It is correctly routed back to your firewall if you are using additional NAT range from your ISP
- It is not used by other device in the network which could cause IP conflict
- If the destination server is using some kind of IP whitelisting if configured to allow your IP address. You mentioned it is "common web server" I would expect that is something that is open for any public addresses
P.S. It is normal to see different sport in c2s from dport in s2c. That is because your NAT rule is probably configured to use "dynamic IP and port" type of source NAT. In this case firewall is using the source port as matching criteria to identify for which session the return traffic belongs to. If FW was not translating the source port and forwarded the original source port, it is very possible that two internal hosts generate traffic with same source port, so firewall will not be able to identify to which host it should forward the reply.
10-11-2022 02:05 AM
Hi @Astardzhiev ,
thanks for your quick reply and long explanation, very instructive.
there are no more additional NAT from our ISP that I am aware of and the IP is not used by any other device in the network.
In fact the public IP shown is used by our whole office to access the Internet and this site is the only one having connectivity issues. The target host is a public website of an electricity firm, reachable from my phone without issues.
I did some packecapture from the FW and indeed I only see packets leaving paloalto and attempts of retransmission. Running out of ideas...
I will ask the local provider to confirm there is no filtering from its side.
10-12-2022 05:05 AM
Hey @JoseCortijo ,
In my experience it is more common the destination server to perform some kind of filtering/allowlisting, rather for the ISP to block you.(In general ISP don't have any benefit to restrict your traffic, while at the same time will want to keep you as customer and provide ways to connecting to all parts of the world).
I am wondering if wrong IP geo-location information is not the problem (if the electricity company allow access only from given region/country). If you check your public NAT IP in sites like https://www.iplocation.net/ do you see correct geo-location information? No need to be exact, at least to show the correct country?
10-13-2022 01:03 AM
Hi @Astardzhiev ,
the ISP did a connection test and confirmed that it is our public IP that is blocked at the server level. I wonder what might be the reason behind it. I checked our public IP on the site you mentioned and it shows Spain.
My issue now is how to reach the technicians behind the domain. in whois everything is hidden and only some telephone numbers are offered. I am imagining a non-technical receptionist picking up the phone and myself trying to explain to her all this issue. 😕
Thanks a lot for your support.
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!