Having issues with ipsectunnel after upgrading FWs(5260s) to 10.1.3

[JJoseph]

Hello Palo Community!

I have a a couple of ipsec tunnels connecting to a cloud vendor providing ERP services to our users. Since we upgraded our FWs to 10.1.3 a couple of weeks ago from 9.1.10, we are having issues with connection slowness, timeouts, ssh session termination, webpage not available etc where users aren't able to connect to any services on the other end. There is no issues with phase1 or 2 (gui status and sh vpn ike,ipsec, debug ike/ipsec commands etc ). This was confirmed by Palo(have a case open with them). When I look at the traffic log, the connections are ageing out, whenever I run the sh vpn flow name command when users can't connect, I see numbers increasing for encap packets and zero for decap. I have done packet captures, other commands (show counter global filter packet-filter yes delta yes) on the FW, sent to Palo and they say that they aren't seeing any packet loss from our side but aren't receiving response from the vendor side. I have to disable and enable the ipsec tunnel each time the connections drop(which is like almost every day) to bring back the connections for the users.

It isn't happening at the same time every day either (which would have indicated some kind of processes running requiring heavy resource utilization or something like that causing this issue or coinciding with rekeying etc ). We compared the phase configs at both sides are they are setup with the same parameters. The other side is fortigate. There are no issues mentioned in the 10.1.3 release  incase this is a bug with this release, nothing mentioned in the 10.1.4 HF release as a fix either. I have a frustrated set of users and management asking me to solve this issue thinking it is a Palo issue(since this happened soon after the FWs were upgraded and it was working fine since the tunnel was created more than 6 months ago). I have 7 other ipsec tunnels without any issues at all working fine.

Anyone else come across anything similar or or have any suggestions that I can try?

Thanks!

[SCantwell_IM]

Hello

So, what does "aged out" mean in the session termination reason?

It means that the FW opened a session for Client and Server to talk, and for whatever reason, either Client or Server stopped communication.  The default session timer is 3600 secs (so for up to 1 hour, remote side stopped transmitting/return vpn traffic to your FW)  Your firewall CANNOT cause these two devices to stop talking, unless there is  security policy, security profile, threat, DoS, Zone Protection, etc type of security restriction to END the session.  This is not what you are seeing.

As you correctly stated.. you see ENCAPSULATION occurring but NO DECAP.  So we cannot decap traffic that we are not receiving.

By tearing the VPN down and restarting it, you are clearing out the SPI session from the VPN, ideally from both sides, and negotiating again.   So, if you look at traffic logs (and bring up the column headers for Packets, Packets Sent, Packets Recevied) you should be able to correctly determine/confirm that the FW is egressing packets out, towards the remote side, but getting zero returns.  If true, then can the FW stop the remote side from transmitting it packets?  I ask again... can your FW actually STOP the OTHER side from transmitting their packets?  The answer is NO.   Now, your FW CAN stop receiving their packets, and this is called a security policy., right?

So, if you have traffic leaving your FW and zero packets recieved by FW from the remote side, you have a ISP/remote network issue that would need to be troubleshoot.  The FW is showing you where the issue is at (meaning... since the cutover, things have been weird... doesnt mean that the FW is  the cause, but we can use our logs and logic to confirm where the issue is NOT (which right now, sound like is it NOT your FW)

[dantequintero]

What was the outcome to the TAC case?