Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Weird routing problem from mgmt interface. Icmp-redirects? If so, how to turn it off?

L2 Linker

I have a problem on a PA500. When it attempts to send traffic to one particular subnet via the management interface, the packets are sent to the wrong place. Instead of going to the default gateway, they go to an ASA. All other subnets route correctly.

 

The PA500 has a very simple config. The management interface is connected directly to a switch. A router is directly configured to the same switch. The Palo's default gateway is the router.   No dynamic routing is configured on the Palo. An ASA is connected to the switch through a vwire on this same unit.

 

The PA500 is only used in vwire mode with the switch-to-ASA vwire used to filter web browsing for the internal users.

 

We noticed that this PA500 stopped getting updates a couple of weeks ago. When we checked why, we could see that DNS lookups were failing (the DNS server is on the problematic subnet). Attempts to ping anything on the problematic subnet from the cli fail. Access from the management interface to all other subnets work just fine.

 

When checking the monitor on the PA500, I can see that the traffic from the management interface to the problem subnet is going across the vwire to the ASA instead of to the router (which is the Palo's default gateway). The destination IP address is right, but the MAC address is the ASA's MAC, not the router's.

 

Looking at the ARP table on the Palo, the correct MAC is present for the IP address of the router.

 

The only mechanism I can think of that would explain this is if the Palo got an icmp-redirect from the router at some point telling it to use the ASA to get to the problem subnet. The router is doing dynamic routing, so it is possible that it termporarily lost a route to the problem subnet, and sent an icmp-redirect to the Palo.

 

I've searched the Palo docs, and I can't find mentions of the management interface supporting icmp-redirects, let alone how to clear it and turn it off.

 

Is there something else that could be causing this issue?

10 REPLIES 10

L6 Presenter

Interesting stuff! Can you do a PCAP from the mgmt interface so that (hopefully) will explain a bit more.  Do you also remember what is changes since the last time updates was working for you?

I have a packet capture from the vwire. The source and destination IP, and source MAC addresses are as expected. The destination MAC is the ASA. Not much else to see. I don't see the management interface as an option in the packet capture filter.

 

I could span the switch port to a PC and wireshark it. I may do that. I do have an open ticket on this, but I'm waiting for a tech to contact me. If I don't hear anything soon, I'll do the port span.

 

This Palo has been in operation for at least a year without configuration changes (other than rule changes and updates). 

 

 

Just follow the article in the "PCAP" link so you can do PCAP on the mgmt interface. You cannot do it from the GUI unfortunately. This problem can be fixed if any other interfaces on PA got an lnternet access using service route option:

 

serv rt.PNG

 

But we want to understand what is going on here :0

noservice.JPG

I have the packet capture. Looking at it in wireshark shows the same as the vwire capture. The ping requests are going to the right IP adress, but the wrong MAC.

 

Here's the Palo's ARP table:

Address HWtype HWaddress Flags Mask Iface
10.160.0.3 ether 00:19:07:70:9c:00 C eth0
10.160.0.4 ether a4:6c:2a:08:68:82 C eth0         (This is the ASA)
10.160.0.6 ether bc:f1:f2:96:d0:42 C eth0
10.160.0.2 ether 00:19:07:28:c8:40 C eth0
10.160.0.1 ether 00:a2:ee:73:51:80 C eth0         (This is the Palo's default gateway)

 

Here's the outbound ping packet as shown in wireshark

 

 

Capture1.JPGNotice the destination MAC address is not the MAC address of the default gateway. It is the MAC address of the ASA.

 

 

Hi Ken,

Is there proxy arp enabled on the ASA?
================================================================
ACE 7.0, 8.0, PCNSE 7

From the CLI you can use the 'tcpdump' command to packetcapture directly on the management interface

 

did you make sure to use the 'show arp management' command (so no dataplane arp information is included)

Tom Piens
PANgurus - Strata specialist; config reviews, policy optimization

L6 Presenter

This is very interesting one l am really curious to know what is causing this :0

AFAIK, A Cisco ASA will only proxy arp for a NAT address. Obviously, the IP address of my router isn't a NAT address in the ASA.

 

Besides, doing a "show arp management dns no" give the following table

 

Address HWtype HWaddress Flags Mask Iface
10.160.0.3 ether 00:19:07:70:9c:00 C eth0
10.160.0.4 ether a4:6c:2a:08:68:82 C eth0
10.160.0.6 ether bc:f1:f2:96:d0:42 C eth0
10.160.0.2 ether 00:19:07:28:c8:40 C eth0
10.160.0.11 ether 00:b0:e1:29:90:08 C eth0
10.160.0.1 ether 00:a2:ee:73:51:80 C eth0

 

Notice that the ARP entry for 10.160.0.1 (the Palo's default gateway) is the router's MAC address.

 

The ASA shows up as the correct IP/MAC address (10.160.0.4)

 

 

So is Palo tech support 🙂

  • 4828 Views
  • 10 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!