Palo Alto VM, Layer 2 bridge (transparent), 802.11q sub interface mac flapping on cisco switch

[JustinWoodman]

Hello everyone,

I have an existing palo alto PA-3550 which we are migrating over to vmware, virtualized version (VM-300), onsite, no cloud.  On the appliance we have two sets of layer 2 interfaces bridged together.  One set is basically a transparent firewall, the other is just marking qos traffic.  Since this device has 20 or so 1 gb ports, each layer 2 interface is connected to port without using 802.1q. Since our internet circuit is a symmetric 1 gb service, it makes no sense to do so.  Everything works fine on the appliance, and has for several years.  The appliance uplinks are basically setup with an cisco access vlan and set the access mode. I know that in this setup that I can do a bridge with a network cable, and the cisco switches would be fine.

On the vm-300, I have port 5(eth 1/5) setup for 802.1q vlan tags and recreated the same ports.  It has four sub interfaces, on one port.  In vmware host the port is setup for all vlans(vlan id 4095).  This basically passes all vlans through that port.

Back to the vm-300 specifics, I have the sub interfaces in their respected VLANs, and same policies as before.  It works fine.  The problem I am having is the  pair of cisco 4500X we use as our core switch, have problems with the vm-300 layer 2 bridges.  With no interfaces define on the cisco switch, I get a lot of mac flaps, and it basically does not work as a bridge. I can't ping across it, maybe one out of twenty five.  With a layer 3 interface define on the cisco switch,  spanning tree will block the port.  With layer 3 interface and spanning tree disabled, the switch is brought to it's knees with a loop, these are dual core switch, the core running spanning tree is at 100%. I have also tried this configuration with 802.1q, just a palo alto interface with no sub interfaces, and vmware the ports hard set to the 903(eth1/7) or 904(eth1/8), and had the same reaction. So, I ask am I doing something wrong, or something I need to check or can you not do layer 2 bridges with vm series? I know it's possible to do ethernet bridges in vmware, as I have a couple running on a virtual VyOs router for disaster recovery purposes. We use it to bridge a cisco vrf that replicates our primary site layer 3 interfaces, and share routes across the bridge.

C4K_EBM-4-HOSTFLAPPING: STANDBY:Host 00:50:56:99:41:29 in vlan 903 is moving from port Te2/1/2 to port Te2/1/1
C4K_EBM-4-HOSTFLAPPING: STANDBY:Host 00:50:56:99:41:29 in vlan 903 is moving from port Te2/1/1 to port Te2/1/2
C4K_EBM-4-HOSTFLAPPING: STANDBY:Host 00:50:56:99:22:01 in vlan 903 is moving from port Te2/1/5 to port Te2/1/2
C4K_EBM-4-HOSTFLAPPING: STANDBY:Host 00:50:56:99:41:29 in vlan 903 is moving from port Te2/1/2 to port Te2/1/1
C4K_EBM-4-HOSTFLAPPING: STANDBY:Host 00:50:56:99:41:29 in vlan 903 is moving from port Te2/1/1 to port Te2/1/2
C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:99:22:01 in vlan 903 is moving from port Te2/1/2 to port Te2/1/5
C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:99:22:01 in vlan 903 is moving from port Te2/1/5 to port Te2/1/2
C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:99:41:29 in vlan 903 is moving from port Te2/1/2 to port Te2/1/1
C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:99:41:29 in vlan 903 is moving from port Te2/1/1 to port Te2/1/2

---------------------
Vlan on the appliance PA-3050:

pvst+ tag rewrite:                    enabled
pvst+ native vlan id:                 1
drop stp:                             disabled
802.1Q PCP pass through:              disabled

total vlan shown :                    2

name                interface         virtual interface
--------------------------------------------------------------------------------
Default                                       ethernet1/3
                                                  ethernet1/4
TW-INTERNET-TP-FIREWALL ethernet1/7
                                                  ethernet1/8

--------------------------------------
Vlan on vm-300.  I have a test bridge on it.

pvst+ tag rewrite:                    enabled
pvst+ native vlan id:                 1
drop stp:                             disabled
802.1Q PCP pass through:              disabled

total vlan shown :                    3

name                interface         virtual interface
--------------------------------------------------------------------------------
Default                                       ethernet1/5.123
                                                  ethernet1/5.223
TW-INTERNET-TP-FIREWALL ethernet1/5.500
                                                  ethernet1/5.503
TP-TESTER                              ethernet1/7.903
                                                  ethernet1/7.904

Thanks,
Justin

[JustinWoodman]

Oops.  I left out an important detail.  Both units, the PA-3050 and VM-300 are running firmware version 8.0.18.

Thanks,

Justin

[JustinWoodman · Accepted solution]

In case anyone else is having an unfun time with a VMWare version Palo Alto firewall, setting it up to running a transparent layer 2 bridge with cisco switches, and fighting loops and spanning tree.  I have finally cracked the egg on this issue.  Let me take a step back first. Our esxi hosts are dual 10g connected to two different cisco 4500x switches.  We are not teaming these link, so one link is active for the vm networks, and we do use the other as backup/standby, and the standby is set active for vmotion and esxi network. So we have one active and one passive physical nic in the vmware nodes, and they are not teamed in any way. In this configuration Vmware's ESXi and VCenter will allow ~Any~ physical nic to participate in the vswitch.  So what happens is you get a nasty broadcast/multicast storm.  It allows the standby nics to send broadcast and multicast traffic to the vswitch, even when it's a standby.  This is the source of all my pain.  Don't worry, there is a really easy fix though.  Change each vmhost advanced setting for Net.ReversePathFwdCheckPromiscfrom zero to a 1.  Then you will need to cycle each of the promiscuous mode off and back on for the vswitch, if its enabled there or the vm network that has the promiscuous mode enable overriding the vswitch default. I recommend putting it in maintenance mode and vmotion  everything off the vmhost prior.  The other fix is to only have one physical nic in the switch or use link teaming like lacp and such.  Now everything works correctly, spanning tree is happy as well. The vmware KB article on this is 59235. https://kb.vmware.com/s/article/59235 

Just in case anyone is new to the vmware palo alto firewalls, it requires promiscuous mode out of the box.  There is a way to disable this requirement, if you like.  In short the vm palo alto makes it's own mac addresses, and they different from the ones that vmware assigns to it. Promiscuous mode allows it to work in this manor. Hope this helps someone. 

Thanks,

Justin Woodman

[JustinWoodman]

Correction or clarification on:

In this configuration Vmware's ESXi and VCenter will allow ~Any~ physical nic to participate in the vswitch.

It should say:

In this configuration Vmware's ESXi and VCenter will allow ~Any~ physical nic to participate in the vswitch regarding broadcast and multicast traffic.