03-07-2017 08:03 AM
I have a problem on a PA500. When it attempts to send traffic to one particular subnet via the management interface, the packets are sent to the wrong place. Instead of going to the default gateway, they go to an ASA. All other subnets route correctly.
The PA500 has a very simple config. The management interface is connected directly to a switch. A router is directly configured to the same switch. The Palo's default gateway is the router. No dynamic routing is configured on the Palo. An ASA is connected to the switch through a vwire on this same unit.
The PA500 is only used in vwire mode with the switch-to-ASA vwire used to filter web browsing for the internal users.
We noticed that this PA500 stopped getting updates a couple of weeks ago. When we checked why, we could see that DNS lookups were failing (the DNS server is on the problematic subnet). Attempts to ping anything on the problematic subnet from the cli fail. Access from the management interface to all other subnets work just fine.
When checking the monitor on the PA500, I can see that the traffic from the management interface to the problem subnet is going across the vwire to the ASA instead of to the router (which is the Palo's default gateway). The destination IP address is right, but the MAC address is the ASA's MAC, not the router's.
Looking at the ARP table on the Palo, the correct MAC is present for the IP address of the router.
The only mechanism I can think of that would explain this is if the Palo got an icmp-redirect from the router at some point telling it to use the ASA to get to the problem subnet. The router is doing dynamic routing, so it is possible that it termporarily lost a route to the problem subnet, and sent an icmp-redirect to the Palo.
I've searched the Palo docs, and I can't find mentions of the management interface supporting icmp-redirects, let alone how to clear it and turn it off.
Is there something else that could be causing this issue?
03-07-2017 08:41 AM - edited 03-07-2017 09:05 AM
Interesting stuff! Can you do a PCAP from the mgmt interface so that (hopefully) will explain a bit more. Do you also remember what is changes since the last time updates was working for you?
03-07-2017 09:58 AM
I have a packet capture from the vwire. The source and destination IP, and source MAC addresses are as expected. The destination MAC is the ASA. Not much else to see. I don't see the management interface as an option in the packet capture filter.
I could span the switch port to a PC and wireshark it. I may do that. I do have an open ticket on this, but I'm waiting for a tech to contact me. If I don't hear anything soon, I'll do the port span.
This Palo has been in operation for at least a year without configuration changes (other than rule changes and updates).
03-07-2017 10:40 AM - edited 03-07-2017 01:08 PM
Just follow the article in the "PCAP" link so you can do PCAP on the mgmt interface. You cannot do it from the GUI unfortunately. This problem can be fixed if any other interfaces on PA got an lnternet access using service route option:
But we want to understand what is going on here :0
03-07-2017 01:33 PM - edited 03-07-2017 01:37 PM
I have the packet capture. Looking at it in wireshark shows the same as the vwire capture. The ping requests are going to the right IP adress, but the wrong MAC.
Here's the Palo's ARP table:
Address HWtype HWaddress Flags Mask Iface
10.160.0.3 ether 00:19:07:70:9c:00 C eth0
10.160.0.4 ether a4:6c:2a:08:68:82 C eth0 (This is the ASA)
10.160.0.6 ether bc:f1:f2:96:d0:42 C eth0
10.160.0.2 ether 00:19:07:28:c8:40 C eth0
10.160.0.1 ether 00:a2:ee:73:51:80 C eth0 (This is the Palo's default gateway)
Here's the outbound ping packet as shown in wireshark
03-15-2017 06:00 PM
03-16-2017 02:32 AM
From the CLI you can use the 'tcpdump' command to packetcapture directly on the management interface
did you make sure to use the 'show arp management' command (so no dataplane arp information is included)
03-16-2017 02:35 AM
This is very interesting one l am really curious to know what is causing this :0
03-22-2017 01:24 PM
AFAIK, A Cisco ASA will only proxy arp for a NAT address. Obviously, the IP address of my router isn't a NAT address in the ASA.
Besides, doing a "show arp management dns no" give the following table
Address HWtype HWaddress Flags Mask Iface
10.160.0.3 ether 00:19:07:70:9c:00 C eth0
10.160.0.4 ether a4:6c:2a:08:68:82 C eth0
10.160.0.6 ether bc:f1:f2:96:d0:42 C eth0
10.160.0.2 ether 00:19:07:28:c8:40 C eth0
10.160.0.11 ether 00:b0:e1:29:90:08 C eth0
10.160.0.1 ether 00:a2:ee:73:51:80 C eth0
Notice that the ARP entry for 10.160.0.1 (the Palo's default gateway) is the router's MAC address.
The ASA shows up as the correct IP/MAC address (10.160.0.4)
03-22-2017 01:24 PM
So is Palo tech support 🙂
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!
