The culporit in this whole thing is HSRP. The design of HSRP is flawed as each vlan has an active and standby hsrp peer. Because the vlan network has an SVI on each HSRP peer regardless of active or standby, the network is known as connected and tells this to each palo alto. So the palo alto sees each downstream HSRP peer as equal cost, so it at times sends return traffic for that said vlan to the standby HSRP peer and I assume the ARP and MAC address as some mismatches doing on since from the clients aspect the gateway address for the HSRP group sends a virtual mac, but when traffic returns the peers use their real mac and not the virtual. So I can fix this by just turning off ECMP on the palo altos which means I will use ecmp from the core 9500s out to the rest of the network but return traffic will only return on one path. BUT what I do not get is that if I am setting my ECMP with symmetric return why would it ingest traffic from the active hsrp peer and not send it back to the same peer as the reason for symmetric return is "return traffic on the same interface it was received on"
... View more