Best config to speed up HA failover

[DraganMilojevic]

During the last PAN OS upgrade we had to failover between two firewalls in HA configuration. The failover time takes unusually amount of time during which the Internet access was unavailable. It took approximately 10-15 lost pings (to internet host) for passive to become an active. We had opened a case with PAN support and our zoom meeting was dropping, it was reconnecting after about 15 sec automatically. In one of my previous jobs the failover was taking very quickly, i would lost 1 or 2 pings 8.8.8.8..

Our HA setup is like this:

HA1 - over aux-1

HA2 - over eth1/10

Mode is active-passive/the config sync is enabled/passive link state is auto/preemptive is not setup/LACP-LLDP is not configured/Link and path monitorings are enabled/

Wondering if someone had simmilar experience and what was the solution to speed up the failover.

[Raido_Rattameister]

In Passive Palo add capture filter that includes non-ip.

If you can't find really calm moment (during offhours) with low traffic volume then choose only one low utilization interface to reduce amount of data logged.

Send retrieve and transit captures into same file.

Start capture.

Send active firewall to reboot.

Wait until traffic starts flowing.

Stop capture.

Analyze how long it takes from Palo sending out gratuitous arp until traffic started flowing in. If it is 15 seconds then go and analyze why it takes switch so long to switch ports active.

If you are inside then you can ping firewall IP then it is easy to filter based on arp and ping packets to identify the delay.

Are there any other packets in between arp going out and ping coming in that might reveal what is going on on the switchport?

[seb_rupik]

Hi @dgsans ,

That time window sounds eerily close to the STP forward delay timer.

Can you share the output from both switches of:

show interface <firewall_uplink> switchport
show spanning-tree vlan <firewall_trunk_vlans>

cheers,

Seb.

[dgsans]

Output of those commands for all three interfaces that one of the firewalls is connected to.  Yes, it does seem to correlate to forward delay timer value.  But if switch ports are in portfast mode, shouldn't it be coming up right away?  If this is a small office with only the one switch, should I just disable STP on the VLANs connected to the firewall or is there an adjustment to be made to the forward, hello, or other timers for STP to get intentional failover to take only 1 second?

Thanks

[seb_rupik]

Hi there,

If the firewall HA Passive link state set to auto? This should set the link state to UP for the passive firewall. The switchport on the other end should be in a Forwarding state if PortFast has been enabled.

Were you able to gather the switch output requested before?

cheers,

Seb.

[dgsans]

Yes, the HA Passive Link State is set to auto.  I see the interfaces as Green on the Dashboard on both firewalls.

Whoops, I forgot to paste the switch output in last reply.  Here it is:

swt-acc#sho interfaces gigabitEthernet 1/0/3 switchport
Name: Gi1/0/3
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 10 (VLAN-10-ISP-INTERNET)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Vepa Enabled: false
App Interface: false
Appliance trust: none

swt-acc#sho interfaces gigabitEthernet 1/0/13 switchport
Name: Gi1/0/13
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 20 (VLAN-20-OFFICE)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Vepa Enabled: false
App Interface: false
Appliance trust: none

swt-acc#sho interfaces gigabitEthernet 1/0/37 switchport
Name: Gi1/0/37
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 30 (VLAN-30-MGMT)
Trunking Native Mode VLAN: 1 (default)
Administrative Native VLAN tagging: enabled
Voice VLAN: none
Administrative private-vlan host-association: none
Administrative private-vlan mapping: none
Administrative private-vlan trunk native VLAN: none
Administrative private-vlan trunk Native VLAN tagging: enabled
Administrative private-vlan trunk encapsulation: dot1q
Administrative private-vlan trunk normal VLANs: none
Administrative private-vlan trunk associations: none
Administrative private-vlan trunk mappings: none
Operational private-vlan: none
Trunking VLANs Enabled: ALL
Pruning VLANs Enabled: 2-1001
Capture Mode Disabled
Capture VLANs Allowed: ALL

Protected: false
Unknown unicast blocked: disabled
Unknown multicast blocked: disabled
Vepa Enabled: false
App Interface: false
Appliance trust: none

swt-acc#show spanning-tree vlan 10

VLAN0010
Spanning tree enabled protocol rstp
Root ID Priority 32768
Address b8f8.5386.0d21
Cost 20004
Port 1 (GigabitEthernet1/0/1)
Hello Time 2 sec Max Age 6 sec Forward Delay 2 sec

Bridge ID Priority 32778 (priority 32768 sys-id-ext 10)
Address 5c31.924e.4c00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1 Root FWD 20000 128.1 P2p
Gi1/0/3 Desg FWD 20000 128.3 P2p Edge
Gi1/0/4 Desg FWD 20000 128.4 P2p Edge

swt-acc#show spanning-tree vlan 20

VLAN0020
Spanning tree enabled protocol rstp
Root ID Priority 32788
Address 5c31.924e.4c00
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32788 (priority 32768 sys-id-ext 20)
Address 5c31.924e.4c00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/13 Desg FWD 20000 128.13 P2p Edge
Gi1/0/14 Desg FWD 20000 128.14 P2p Edge
Gi1/0/17 Desg FWD 20000 128.17 P2p Edge

swt-acc#show spanning-tree vlan 30

VLAN0030
Spanning tree enabled protocol rstp
Root ID Priority 32798
Address 5c31.924e.4c00
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32798 (priority 32768 sys-id-ext 30)
Address 5c31.924e.4c00
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/37 Desg FWD 20000 128.37 P2p Edge
Gi1/0/38 Desg FWD 20000 128.38 P2p Edge
Gi1/0/39 Desg FWD 20000 128.39 P2p Edge
Gi1/0/40 Desg FWD 20000 128.40 P2p Edge
Gi1/0/41 Desg FWD 20000 128.41 P2p Edge

[seb_rupik]

Hi there, 

Thanks for the output. Can I assume that Gi1/0/4 connects to the other firewall in the cluster? What it connected to Gi1/0/1 ? Why is that the root-bridge for VLAN10 ?

It would be interesting to run debug spanning-tree event on the switch during a firewall failover to see what the switch thinks is happening? 

I don't think you've given the detail, but is this a 'U' shape deployment, with each firewall in the cluster connected to a different switch and a cross link between the switches? The switches are logically separate?

cheers,

Seb.

[dgsans]

Its a single switch that will be used for a small office setup.

Ports 1-6 are set to VLAN 10 for "INTERNET" access.

- port 1 currently goes to another Cisco switch that is part of my home lab setup -- it in turn is behind my FiOS router.

     - that is probably why this is the root bridge since it links to another switch.  There is only a single connection to the upstream home lab switch.

- port 3 is the external interface of the primary PA

- port 4 is the external interface of the secondary PA

Ports 13-36 are for VLAN 20 for OFFICE devices

- port 13 is the internal interface of the primary PA

- port 14 is the internal interface of the secondary PA

Ports 37-42 are for VLAN 30 for MANAGEMENT devices

- port 37 is the eth1/3 interface of the primary PA connecting it to the management network

- port 38 is the eth1/3 interface of the secondary PA connecting it to the management network

- ports 39 and 40 are the OOB management ports on the two PAs

If the direct linkage of two switches is potentially causing delay issues, I suppose I could configure two ports on the home lab switch fr the correct VLAN and connect the external interface (ethernet1/1) on each PA to it directly ... thus the switch behind the PAs would only be connected to end devices, no other switches.

[seb_rupik]

Hi there,

I don't believe having the root-bridge for VLAN10 else where is causing your problem. I was just trying to picture your topology.

I you could share the debug output during a failover event, that would be a great next step.

cheers,

Seb.

[mcftang]

Here is the only way if you are using Active-Passive Mode.   To totally disable STP with these two commands together.  Around 5 seconds in switchover process if no STP process on switch.

spanning bpdufilter enable

spanning portfast trunk

For VirtualWire and Layer2 Mode, you may have problem when using Passive link state auto and Network pre-negotiation enabled because PA program is crazy.  During a SUSPEND/REBOOT/SHUTDOWN, it will activate the passive node for around 0.5 second before deactivating the crrent activate node.  That means, it will LOOP.  If you have multicast applications in your network, you are done.  You have to use your owned way to avoid that 0.5 second loop.  So, Passive link state auto is meaningless on PA firewall.

For Virtual Wire, you may try Active-Active mode and enable everything, then, use rapid-pvst or MST on Cisco switches.  They means you loop the network always (besides PA crazy program 0.5 second loop) and using RSTP to break the loop.  So, the downtime dring switch over is in second level.  For L2 Mode, no way (Active-Active not allow).