topic Re: NGFW 1400 Series LACP / Failover issue (11.1.5) in Next-Generation Firewall Discussions

NGFW 1400 Series LACP / Failover issue (11.1.5)

Z.Boussaid — Wed, 08 Jan 2025 12:58:08 GMT

This is a notification for anyone running 1420 boxes in a high-availability (HA) configuration in their environments >11.1.4. We recently encountered significant issues with our NGFW operating in HA mode. Specifically, the HA setup failed on the active firewall, and the failover did not occur as expected to the secondary (standby) device. I had to manually suspend the active firewall to enable the secondary to take over, resulting in a brief but severe outage.

A ticket has been opened with TAC, and it appears we are among the first customers to report this issue. TAC has confirmed it to be an LACP-related bug that originated in version 11.1.4. Unfortunately, even downgrading to the preferred version 11.1.4-h7 does not resolve the problem. According to TAC, this issue is still under internal investigation and has not yet been made public. Their senior engineers are actively working on it.

As a temporary workaround, I configured LACP to passive mode between the firewall and the core switch. So far, this adjustment has stabilized the HA setup, with both firewalls operating in Active/Standby mode without errors.

If you encounter a similar issue, provide TAC with the following reference issue: PAN-275888.

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

Z.Boussaid — Mon, 13 Jan 2025 20:38:13 GMT

TAC confirmed they introduced a new feature to help with performance in conditions where the packet rate is low, however this "feature" has an impact when packets spiked causing the data plane to crash. they recommend upgrading to 11.1.6 (not preferred) to fix issue PAN-263208.

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

SanilHande — Thu, 16 Jan 2025 06:44:27 GMT

Hi , Thanks for sharing

Looks like for us its LACP issue (we encountered an issue where the firewall was unable to learn the MAC address from the core switch and all the services found unreachable after upgrading from 11.0.3-h10 to 11.1.2-h15 )

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

Z.Boussaid — Fri, 14 Feb 2025 14:46:28 GMT

we had multiple interface failures, so even after upgrading to 11.1.6 and RMA the box, we just had another similar crash 02-13. all interfaces (OOB, Inside and DMZ) went down, causing of course the OSPF to fail and therefore no internet for all users. the failover didn't happen, even though the HA is enabled. we upgraded the ticket to critical and TAC are still trying to figure out this garbage.

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

aaronbennett — Sun, 16 Feb 2025 23:44:57 GMT

We had the same issue this morning. Lacp had failed from core cisco 9500 switch to palo 1420 to primary firewall. The ha firewall never failed over and I had to reboot the firewall get all layer 3 connectivity back up. I have a case open with tac and they are researching tsf. On preferred code version 11.1.4h7. Waiting for an update and wondering if this same potential bug can effect our 5260 firewalls to cause lacp failure.

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

SanilHande — Mon, 17 Feb 2025 09:10:27 GMT

There is a lot common for this LACP issue 1420 model , We got below recommendation for PA TAC for upgrade issue ?

************************************************************************************************************

Root Cause:
The issue is triggered when the HA device has a ha_group_id set to nonzero, causing both devices to have the same system MAC address, regardless of whether the configuration option "Same System MAC Address for Active-Passive HA" is disabled or not.

Conditions to trigger the issue: HA A/P is enabled, LACP is enabled on AE interfaces with Enable in HA Passive State' is enabled and 'Same System MAC Address for Active-Passive HA' is disabled

To verify this, you can run the command:
show lacp aggregate-ethernet all

If the System MAC address appears the same on both devices, this confirms the issue. We have also successfully reproduced this in our lab environment.

Issue Details & Fix Versions:
Issue ID: PAN-278296
Fix Versions:
11.1.8 – March 6, 2025
11.2.8 – May 5, 2025
11.1.11 – June 6, 2025
12.1.2 – (Date TBD)

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

Z.Boussaid — Mon, 17 Feb 2025 13:02:30 GMT

We recently encountered the same issue even after replacing the device through an RMA. As a result, we escalated the support case to a Critical priority, prompting involvement from Palo Alto Engineering, as the problem began impacting production during peak hours. The most recent incident occurred on February 13, 2025.

Following their investigation, TAC confirmed that the root cause of the PA crash is related to processing a DNS over HTTPS (DoH) packet. This feature was introduced in PAN-OS 11.0 and is enabled by default in PAN-OS 11.0.x. This explanation aligns with our environment, as we run DNS Security on our Edge PA but not on our SD-WAN PA, despite both operating on the same software version.

According to TAC, this issue was expected to be resolved in PAN-OS 11.1.6, which we upgraded to in January. However, the problem persists. If you are running the DNS Security license on your firewalls, it is recommended to execute the specified command on both devices in an HA pair until the release of the PAN-OS 11.1.8 fix, scheduled for the first week of March. Ensure you save your configuration as a precaution and verify with TAC whether DNS Security is enabled in your environment before proceeding.

Additionally, regarding the failover issue, TAC confirmed that it did not occur because the system MAC address remains the same on both devices, as previously discussed in this thread. You can run the second command to verify this. This issue is also expected to be addressed in PAN-OS 11.1.8.

set deviceconfig setting dns-over-https enable no

show lacp aggregate-ethernet all

Re: NGFW 1400 Series LACP / Failover issue (11.1.5)

J.LowZheTing — Tue, 25 Nov 2025 06:53:46 GMT

I'm running PA1420 Active/Passive HA with LACP interface. The firmware is 11.2.4h. FW2 is active and FW1 is passive. When failover to FW1, the LACP is not working and connection is not established. After reverted to FW2 active firewall, all are working fine. The FW1 system log show as below.

Just curious that anyone have the same issue like mine. Please share your opinion for this.

2025/11/24 19:42:57 high ha state-change 0 HA Group 1: Moved from state Passive to state Active
2025/11/24 19:42:57 info routing routed-fib-sync-sel 0 FIB HA sync started when local device becomes master.
[7mmore[27m[K
[K2025/11/24 19:42:58 info iot icd-ha-status 0 Icd HA state is changed from 0 to 1 time: 2025-11-24 19:42:59
2025/11/24 19:42:58 info iot icd-ha-status 0 Icd HA better state is changed from 2450 to 3344 time: 2025-11-24 19:42:59
2025/11/24 19:42:58 info userid dsc-ha-status 0 dsc HA state is changed from 0 to 1 time: 2025-11-24 19:42:59
2025/11/24 19:42:58 info userid 10.1.3 connect-ldap-sever 0 ldap cfg LDAP-Group-Mapping connected to server 10.1.30.148:389, initiated by: 10.1.30.201
2025/11/24 19:43:17 info ctd-agent ctd-agent-connectio 0 Connection to ADNS service is disconnected
2025/11/24 19:43:29 info ctd-agent ctd-agent-connectio 0 Connection to AdnsTelemetry service is disconnected
2025/11/24 19:43:29 info ras rasmgr-ha-full-sync 0 RASMGR daemon sync all user info to HA peer started.
2025/11/24 19:43:29 info vpn keymgr-ha-full-sync 0 KEYMGR sync all IPSec SA to HA peer started.
2025/11/24 19:43:29 info satd satd-ha-full-sync-s 0 SATD daemon sync all gateway infos to HA peer started.
2025/11/24 19:43:29 info routing routed-fib-sync-pee 0 FIB HA sync started when peer device becomes passive.
2025/11/24 19:43:29 info ras rasmgr-ha-full-sync 0 RASMGR daemon sync all user info to HA peer exit.
2025/11/24 19:43:30 info iot icd-ha-status 0 Icd HA better state is changed from 3344 to 3345 time: 2025-11-24 19:43:31
2025/11/24 19:43:30 info userid 10.1.3 connect-ldap-sever 0 ldap cfg LDAP-Group-Mapping connected to server 10.1.30.148:389, initiated by: 10.1.30.201
2025/11/24 19:43:30 info vpn keymgr-ha-full-sync 0 KEYMGR sync all IPSec SA to HA peer exit.
2025/11/24 19:43:32 medium wildfire wildfire-conn-faile 0 Failed to resolve host wildfire.paloaltonetworks.com
2025/11/24 19:44:21 critical lacp ethern unresponsive 0 LACP interface ethernet1/7 moved out of AE-group ae8(peer is not responding to new LACP connection)
2025/11/24 19:44:22 critical lacp ethern unresponsive 0 LACP interface ethernet1/10 moved out of AE-group ae8(peer is not responding to new LACP connection)
2025/11/24 19:44:22 critical lacp ethern unresponsive 0 LACP interface ethernet1/8 moved out of AE-group ae8(peer is not responding to new LACP connection)
2025/11/24 19:44:22 critical lacp ethern unresponsive 0 LACP interface ethernet1/9 moved out of AE-group ae8(peer is not responding to new LACP connection)
2025/11/24 19:44:25 critical lacp ethern unresponsive 0 LACP interface ethernet1/5 moved out of AE-group ae8(peer is not responding to new LACP connection)
2025/11/24 19:44:25 critical lacp ethern unresponsive 0 LACP interface ethernet1/6 moved out of AE-group ae8(peer is not responding to new LACP connection)
2025/11/24 19:44:31 critical lacp ethern unresponsive 0 LACP interface ethernet1/17 moved out of AE-group ae1(peer is not responding to new LACP connection)
2025/11/24 19:44:32 critical lacp ethern unresponsive 0 LACP interface ethernet1/18 moved out of AE-group ae1(peer is not responding to new LACP connection)