topic Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off? in General Topics

Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

Ken_Cornetet — Tue, 07 Mar 2017 16:03:16 GMT

I have a problem on a PA500. When it attempts to send traffic to one particular subnet via the management interface, the packets are sent to the wrong place. Instead of going to the default gateway, they go to an ASA. All other subnets route correctly.

The PA500 has a very simple config. The management interface is connected directly to a switch. A router is directly configured to the same switch. The Palo's default gateway is the router. No dynamic routing is configured on the Palo. An ASA is connected to the switch through a vwire on this same unit.

The PA500 is only used in vwire mode with the switch-to-ASA vwire used to filter web browsing for the internal users.

We noticed that this PA500 stopped getting updates a couple of weeks ago. When we checked why, we could see that DNS lookups were failing (the DNS server is on the problematic subnet). Attempts to ping anything on the problematic subnet from the cli fail. Access from the management interface to all other subnets work just fine.

When checking the monitor on the PA500, I can see that the traffic from the management interface to the problem subnet is going across the vwire to the ASA instead of to the router (which is the Palo's default gateway). The destination IP address is right, but the MAC address is the ASA's MAC, not the router's.

Looking at the ARP table on the Palo, the correct MAC is present for the IP address of the router.

The only mechanism I can think of that would explain this is if the Palo got an icmp-redirect from the router at some point telling it to use the ASA to get to the problem subnet. The router is doing dynamic routing, so it is possible that it termporarily lost a route to the problem subnet, and sent an icmp-redirect to the Palo.

I've searched the Palo docs, and I can't find mentions of the management interface supporting icmp-redirects, let alone how to clear it and turn it off.

Is there something else that could be causing this issue?

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

TranceforLife — Tue, 07 Mar 2017 17:05:34 GMT

Interesting stuff! Can you do a PCAP from the mgmt interface so that (hopefully) will explain a bit more. Do you also remember what is changes since the last time updates was working for you?

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

Ken_Cornetet — Tue, 07 Mar 2017 17:58:49 GMT

I have a packet capture from the vwire. The source and destination IP, and source MAC addresses are as expected. The destination MAC is the ASA. Not much else to see. I don't see the management interface as an option in the packet capture filter.

I could span the switch port to a PC and wireshark it. I may do that. I do have an open ticket on this, but I'm waiting for a tech to contact me. If I don't hear anything soon, I'll do the port span.

This Palo has been in operation for at least a year without configuration changes (other than rule changes and updates).

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

TranceforLife — Tue, 07 Mar 2017 21:08:39 GMT

Just follow the article in the "PCAP" link so you can do PCAP on the mgmt interface. You cannot do it from the GUI unfortunately. This problem can be fixed if any other interfaces on PA got an lnternet access using service route option:

But we want to understand what is going on here :0

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

Ken_Cornetet — Tue, 07 Mar 2017 21:03:11 GMT

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

Ken_Cornetet — Tue, 07 Mar 2017 21:37:24 GMT

I have the packet capture. Looking at it in wireshark shows the same as the vwire capture. The ping requests are going to the right IP adress, but the wrong MAC.

Here's the Palo's ARP table:

Address HWtype HWaddress Flags Mask Iface
10.160.0.3 ether 00:19:07:70:9c:00 C eth0
10.160.0.4 ether a4:6c:2a:08:68:82 C eth0 (This is the ASA)
10.160.0.6 ether bc:f1:f2:96:d0:42 C eth0
10.160.0.2 ether 00:19:07:28:c8:40 C eth0
10.160.0.1 ether 00:a2:ee:73:51:80 C eth0 (This is the Palo's default gateway)

Here's the outbound ping packet as shown in wireshark

Notice the destination MAC address is not the MAC address of the default gateway. It is the MAC address of the ASA.

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

ansharma — Thu, 16 Mar 2017 01:00:49 GMT

Hi Ken,

Is there proxy arp enabled on the ASA?

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

reaper — Thu, 16 Mar 2017 09:32:54 GMT

From the CLI you can use the 'tcpdump' command to packetcapture directly on the management interface

did you make sure to use the 'show arp management' command (so no dataplane arp information is included)

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

TranceforLife — Thu, 16 Mar 2017 09:35:18 GMT

This is very interesting one l am really curious to know what is causing this :0

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

Ken_Cornetet — Wed, 22 Mar 2017 20:24:19 GMT

AFAIK, A Cisco ASA will only proxy arp for a NAT address. Obviously, the IP address of my router isn't a NAT address in the ASA.

Besides, doing a "show arp management dns no" give the following table

Address HWtype HWaddress Flags Mask Iface
10.160.0.3 ether 00:19:07:70:9c:00 C eth0
10.160.0.4 ether a4:6c:2a:08:68:82 C eth0
10.160.0.6 ether bc:f1:f2:96:d0:42 C eth0
10.160.0.2 ether 00:19:07:28:c8:40 C eth0
10.160.0.11 ether 00:b0:e1:29:90:08 C eth0
10.160.0.1 ether 00:a2:ee:73:51:80 C eth0

Notice that the ARP entry for 10.160.0.1 (the Palo's default gateway) is the router's MAC address.

The ASA shows up as the correct IP/MAC address (10.160.0.4)

Re: Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

Ken_Cornetet — Wed, 22 Mar 2017 20:24:53 GMT

So is Palo tech support 🙂