A/A vWire Deployment Forwarding MAC Address on HA Links?

Reply
Highlighted
L1 Bithead

A/A vWire Deployment Forwarding MAC Address on HA Links?

Hey Guys,

 

I'm having an odd MAC flapping issue when I implemented a A/A PAN under a A/P ASA. I'll give the high level and attach a topology with the failure patterns I saw.

 

We have a pair of 5585X's as the traditional L3 / L4 internet facing Firewall. We were looking to update our threat prevention architecture, remove some inline taps and consolidate feature into a unified platform and PAN was picked. Due to the network topology the best place for us to put the PAN's is under the ASA on the "Inside" zone.

 

The PAN's are deployed in vWire for full UTM, Web Filtering and Decrypt. It's a pair of 5260's in A/A is deployed using AUX Ports as HA1/HA1b, 2 physical ports as HA2/HA2b, and 2 physicals aggregated as HA3. 

 

Physical topology looks like this

 

PAN Boards.png

 

This was a phased rollout where Rack 1 was done first and Rack 2 was done a week later, Rack 1 was completed with no issue. When we inserted the PAN in Rack 2  we were seeing the MAC address of the Active ASA interface learned from 2 ports on the Inside Switch and getting MAC Flap errors. Its caused complete chaos and we had to roll back. The only logical path I can gather is that it was coming through the HA link on the PAN.

 

We also have a SSL VPN DMZ zone that was configured the same way thorugh a different vWire that DIDN'T see this MAC FLAP issue, that topology can be seen here:

 

PAN Boards 1.png

Lastly, I've recreated this enviornment in my LAB (PAN 820's) and I'm unable to replicate the MAC flapping issue I see in production.

 

Anyone have any ideas? If you need more information let me know and I can get it pretty quickly.

Highlighted
L2 Linker

Re: A/A vWire Deployment Forwarding MAC Address on HA Links?

ASA did no failover/split-brain during this change?

Highlighted
L1 Bithead

Re: A/A vWire Deployment Forwarding MAC Address on HA Links?

No it didnt, the the ASA show failover was clean the second FW was in standby ready state with all expected interfaces in Normal / Monitored state.

Highlighted
L2 Linker

Re: A/A vWire Deployment Forwarding MAC Address on HA Links?

In vWire, packets will (according to documentation) always leave the same box which received the packet. If the other box is responsible for L7 processing the packet will be sent trough HA3 to the other box for L7 processing. After this, it is either blocked or sent back to the first box trough HA3 which will send it out the other side of the vWire. In no case (bugs excluded), a "vWire"-packet will leave the data interface on the other box. With this, it is guaranteed Paloalto wont generate a loop since it behaves like a cable.

 

Sorry, currently no clue why this could be happened. 

Highlighted
L1 Bithead

Re: A/A vWire Deployment Forwarding MAC Address on HA Links?

No problem, I'm working through a case with Support and the the SE. Whatever the outcome is its going to either be interesting or really stupid. Fully mocked up in the lab I cant get it to do the same, only difference is 820's in lab vs 5260's in production!

Highlighted
L1 Bithead

Re: A/A vWire Deployment Forwarding MAC Address on HA Links?

Just to give an update in case anyone runs across this thread.

 

This is a severe bug thats been identified in the 5200 platform. In specific scenarios when an A/A vWire topology is used we are seeing packets loose the A/A Device ID # from the session headers. This is creating a situation where packets are being forwarded out a vWire from the firewall that didnt originate the session. Hence the Mac Flapping we were seeing in the southbound switches.

 

I will post the bug ID once PAN makes it public. The solution will be to upgrade code that has the software fix in it.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!