topic Re: Best config to speed up HA failover in General Topics

Best config to speed up HA failover

DraganMilojevic — Mon, 29 Jul 2019 15:44:05 GMT

During the last PAN OS upgrade we had to failover between two firewalls in HA configuration. The failover time takes unusually amount of time during which the Internet access was unavailable. It took approximately 10-15 lost pings (to internet host) for passive to become an active. We had opened a case with PAN support and our zoom meeting was dropping, it was reconnecting after about 15 sec automatically. In one of my previous jobs the failover was taking very quickly, i would lost 1 or 2 pings 8.8.8.8..

Our HA setup is like this:

HA1 - over aux-1

HA2 - over eth1/10

Mode is active-passive/the config sync is enabled/passive link state is auto/preemptive is not setup/LACP-LLDP is not configured/Link and path monitorings are enabled/

Wondering if someone had simmilar experience and what was the solution to speed up the failover.

Re: Best config to speed up HA failover

OtakarKlier — Mon, 29 Jul 2019 16:05:22 GMT

Hello,

This is the best practice for upgrades. Hopefully you didnt just reboot the firewall and instead used the 'Ssupend' Feature.

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClRrCAK

Regards,

Re: Best config to speed up HA failover

Retired Member — Mon, 29 Jul 2019 16:41:25 GMT

Do you use pppoe for the internet connection?

Re: Best config to speed up HA failover

DraganMilojevic — Mon, 29 Jul 2019 18:32:20 GMT

No, it is a "normal" ISP connection, 50Mbps

Re: Best config to speed up HA failover

DraganMilojevic — Mon, 29 Jul 2019 18:33:42 GMT

Appreciate your concern; i have been working with PAN for a quite some time and never had an issue with OS upgrade but that was not my question...

Re: Best config to speed up HA failover

Sec101 — Tue, 30 Jul 2019 02:35:58 GMT

this sounds like a spanning-tree issue- the time it takes for that port to come up - could be STP going through the listening learning .....stages

Re: Best config to speed up HA failover

BPry — Tue, 30 Jul 2019 02:56:50 GMT

@DraganMilojevic,

What @OtakarKlier was rightfully pointing out was that the proper upgrade procedure would likely have prevented any extended failover outage, as it's a 'clean' way of switching active status to the other peer. If you simply restarted the firewall as part of the upgrade procedure without suspending it, there are a variety of settings that could cause an extended period of time to elapse before traffic starts flowing through the peer unit. If this is the case, we could actually recommend looking at different log files instead of looking for configuration / configuration issues that would cause extended failover time.

There are a variety of settings that I would look at to narrow it down. The first being if LACP aggregates are in use at all, then the HA timer settings deployed on the device, that STP is setup correctly on your switches (took me to long to type this reply, +1 to @Sec101 for being technically the first person to bring up STP), and lastly what @Retired Member mentioned with his PPOE suggestion. You've already said no to two of the four, so the remaining two are things you should look into.

FYI, your comment to @OtakarKlier came off a little rude. Please keep in mind that some suggestions or comments, when answered, will lead to your solution. As mentioned earlier, the order that you performed the upgrade is actually highly important in knowing where we should actually be looking for issues. So if you didn't actually follow recommended procedures, we kinda need to know about it so we don't send you down a rabbit hole troubleshooting the wrong thing.

Unless someone has the title 'Community Manager', everyone that comments on this post is devoting time out of our day to help you answer a question/problem you are having. Please consider that when responding to someone spending part of their day helping others on this forum.

Re: Best config to speed up HA failover

DraganMilojevic — Tue, 30 Jul 2019 14:45:32 GMT

Thanks Sec101, appreciate your comment; i dont think the issue is STP since firewalls are conencted directly; no switch in between.

Re: Best config to speed up HA failover

DraganMilojevic — Tue, 30 Jul 2019 15:06:50 GMT

Thanks BPry, appreciate your comment and time spent answering question.
The 'power off' upgrade process was less likely applicable for my case as i mentioned in the original post that during my previous jobs i had successfully performed PAN OS upgrade, losing 1,2 pings which makes me believe i did follow the upgrade process properly and not just powering off the firewall. I would suspect that powering off the firewall will cause more lost pings.
I am leaning more towards LACP settings at this moment.
I am strong believer that this community is a great place to get answer, sharing ideas, best practice, tips and trick and that everyone's time is valuable, including mine. Being in this line of work for quite some time, i do understand the importance of right information so i am trying to put as much useful details in the original post as possible which should allow people willing to assist to be pointed into right direction. If there is not enough information, it is much easier to ask question instead making assumptions. I think everyone will benefit from this..

Re: Best config to speed up HA failover

OtakarKlier — Tue, 30 Jul 2019 18:27:14 GMT

Hello,

Also in version 8 you can modify the parameters that are used to speed this up a bit. Device tab->high availability->General tab->election settings. I would say set them to aggressive and give it a test.

Regards,

Re: Best config to speed up HA failover

DraganMilojevic — Wed, 07 Aug 2019 15:47:25 GMT

Thanks, i will try that in my next maintenance window.

Cheers

Re: Best config to speed up HA failover

jeremy.larsen — Wed, 07 Aug 2019 16:24:47 GMT

I'm going to concur with sec101 on this. If your switch ports are not set to go straight into the forwarding state, you may have delays while STP goes through it's learning process. 15s sounds about right. If you are using Cisco equipment, that's the default learning timer. I'm not sure how this is affected by Passive Link State: auto, but you should probably have these ports in "portfast" mode (Cisco speak) or "edge" mode (Juniper speak) either way.

https://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/19120-122.html

forward delay—The forward delay is the time that is spent in the listening and learning state. This time is equal to 15 sec by default, but you can tune the time to be between 4 and 30 sec.

PS - Also agree with using "suspend" instead of just reboot for HA failover. Upgrade the Passive node first thus reducing your failover to a single event as well.

Re: Best config to speed up HA failover

dgsans — Wed, 29 Mar 2023 21:00:28 GMT

Dragan, wondering if you ever got a solution for this issue?

We are having the same. I have a new pair of PA-440s with 8 Ethernet ports plus one management port. There are no dedicated HA ports on the model.

Firewalls are currently running PAN-OS 10.2.3-h4.

Eth1/1 is the external interface

Eth1/2 is the internal network

Eth1/3 is the management network

Eth1/4 is unused

Eth1/5 is used for HA1

Eth1/6 is used for HA1 Backup

Eth1/7 is used for HA2

Eth1/8 is used for HA2 Backup

HA setup is active/passive

The management port goes to a switchport configured for the management VLAN.

There is a Cisco Catalyst 9300 switch. Three VLANs are configured for EXTERNAL, INTERNAL and MANAGEMENT.

All the switch ports that the PA firewalls are connected to are set to portfast mode.

I have a constant ping running from a computer behind the firewall to 8.8.8.8

When I go to the active firewall to Device, High Availability, Operational Commands and suspend it, I lose ping responses for close to 30 seconds and then it recovers.

The HA election settings were initially set to Recommended. I set them to Aggressive on both, committed the changes, and suspended the current active one -- still 30 second data loss. I went to Aggressive mode and was going to try setting some values lower, but several of them are apparently at their minimum value in Advanced mode.

No aggregate ports are configured on the firewalls, no LACP is in use -- all connections are singletons.

Config sync is on

Anyone have any suggestions?

Re: Best config to speed up HA failover

Raido_Rattameister — Thu, 30 Mar 2023 00:54:55 GMT

30 second data loss is definitely configuration issue.

Check your switch configs (spanning tree, LACP etc).

HA default timers will give you 0-1 lost ping during failover. No need to adjust HA to aggressive mode.

By default passive will keep it's interfaces in shut mode and this can take time until switch will enable interfaces.

Auto mode gives faster failover as swithport is already up but without knowing config on switch side I can't give any recommendations.

Re: Best config to speed up HA failover

dgsans — Thu, 30 Mar 2023 13:00:48 GMT

I did find one config setting to improve on the PAs. Under Device, High Availability, General, Active/Passive Settings, I had Passive Link State set to Shutdown -- I changed this to Auto on both firewalls, committed and tried again. Now its about 12-13 seconds of data loss during the switchover.

I am not using LACP on the switch -- each firewall has one connection to one switch for each of the three VLANs.

All the switch ports that the firewalls are connected to have portfast enabled. Are there other spanning tree related configurations to check on a Cisco switch when set for portfast?

Thanks

Re: Best config to speed up HA failover

Raido_Rattameister — Fri, 31 Mar 2023 03:32:13 GMT

In Passive Palo add capture filter that includes non-ip.

If you can't find really calm moment (during offhours) with low traffic volume then choose only one low utilization interface to reduce amount of data logged.

Send retrieve and transit captures into same file.

Start capture.

Send active firewall to reboot.

Wait until traffic starts flowing.

Stop capture.

Analyze how long it takes from Palo sending out gratuitous arp until traffic started flowing in. If it is 15 seconds then go and analyze why it takes switch so long to switch ports active.

If you are inside then you can ping firewall IP then it is easy to filter based on arp and ping packets to identify the delay.

Are there any other packets in between arp going out and ping coming in that might reveal what is going on on the switchport?

Re: Best config to speed up HA failover

seb_rupik — Fri, 31 Mar 2023 11:05:50 GMT

Hi @dgsans ,

That time window sounds eerily close to the STP forward delay timer.

Can you share the output from both switches of:

show interface <firewall_uplink> switchport show spanning-tree vlan <firewall_trunk_vlans>

cheers,

Seb.

Re: Best config to speed up HA failover

dgsans — Tue, 04 Apr 2023 19:38:53 GMT

Output of those commands for all three interfaces that one of the firewalls is connected to. Yes, it does seem to correlate to forward delay timer value. But if switch ports are in portfast mode, shouldn't it be coming up right away? If this is a small office with only the one switch, should I just disable STP on the VLANs connected to the firewall or is there an adjustment to be made to the forward, hello, or other timers for STP to get intentional failover to take only 1 second?

Thanks

Re: Best config to speed up HA failover

seb_rupik — Wed, 05 Apr 2023 10:16:46 GMT

Hi there,

If the firewall HA Passive link state set to auto? This should set the link state to UP for the passive firewall. The switchport on the other end should be in a Forwarding state if PortFast has been enabled.

Were you able to gather the switch output requested before?

cheers,

Seb.

Re: Best config to speed up HA failover

dgsans — Wed, 05 Apr 2023 12:10:11 GMT

Yes, the HA Passive Link State is set to auto. I see the interfaces as Green on the Dashboard on both firewalls.

Whoops, I forgot to paste the switch output in last reply. Here it is:

swt-acc#sho interfaces gigabitEthernet 1/0/3 switchport
Name: Gi1/0/3
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 10 (VLAN-10-ISP-INTERNET)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Vepa Enabled: false
App Interface: false
Appliance trust: none

swt-acc#sho interfaces gigabitEthernet 1/0/13 switchport
Name: Gi1/0/13
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 20 (VLAN-20-OFFICE)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Vepa Enabled: false
App Interface: false
Appliance trust: none

swt-acc#sho interfaces gigabitEthernet 1/0/37 switchport
Name: Gi1/0/37
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 30 (VLAN-30-MGMT)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Vepa Enabled: false
App Interface: false
Appliance trust: none

swt-acc#show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32768
Address b8f8.5386.0d21
Cost 20004
Port 1 (GigabitEthernet1/0/1)
Hello Time 2 sec Max Age 6 sec Forward Delay 2 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 5c31.924e.4c00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Root FWD 20000 128.1 P2p
Gi1/0/3 Desg FWD 20000 128.3 P2p Edge
Gi1/0/4 Desg FWD 20000 128.4 P2p Edge

swt-acc#show spanning-tree vlan 20

VLAN0020
Spanning tree enabled protocol rstp
Root ID Priority 32788
Address 5c31.924e.4c00
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32788 (priority 32768 sys-id-ext 20)
Address 5c31.924e.4c00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/13 Desg FWD 20000 128.13 P2p Edge
Gi1/0/14 Desg FWD 20000 128.14 P2p Edge
Gi1/0/17 Desg FWD 20000 128.17 P2p Edge

swt-acc#show spanning-tree vlan 30

VLAN0030
Spanning tree enabled protocol rstp
Root ID Priority 32798
Address 5c31.924e.4c00
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32798 (priority 32768 sys-id-ext 30)
Address 5c31.924e.4c00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/37 Desg FWD 20000 128.37 P2p Edge
Gi1/0/38 Desg FWD 20000 128.38 P2p Edge
Gi1/0/39 Desg FWD 20000 128.39 P2p Edge
Gi1/0/40 Desg FWD 20000 128.40 P2p Edge
Gi1/0/41 Desg FWD 20000 128.41 P2p Edge