arp not found

[AndrewMoore]

Yesterday I attempted to move our Internet connection from a copper interface on ethernet1/1 to fiber optic on ethernet 1/13 on a Palo Alto 3020. 

I ensured both interfaces were members of the  same security zone and modified the Default route of default-vr to use ethernet 1/13 instead of 1/1.

While I and the NOC technician confirmed the fiber port was configured and up, I could not ping out to the Internet. 

During that time ran a show interfaces ethernet1/13 and found 13 instances of "arp not found" under the Logical interface counters read from CPU. 

It seems that maybe I needed to clear the arp table? I say so because I believe the firewall was trying  to reach the MAC of our next hop ( our fiber switch ) through ethernet1/1 instead of ethernet1/13.

Any advice would be greatly appreciated!

[reaper · Accepted solution]

was the next-hop IP identical for the fiber as what it was on copper ?

you could try clearing the arp cache with  

> clear arp ethernet1/1 

to verify if the issue's related to arp, you can check the global counters for more detailed information:

> show counter global filter delta yes | match arp

[Roby_Sreejith]

Have you changed the zones? like ethernet 1/1 zone name and ethernet 1/13 zone?

If same zone you are using, no issue.

otherwie security rule should be created

[AndrewMoore]

Thanks for the reply @Roby_Sreejith. I confirmed before the switch over that the new fiber port ethernet1/13 had been assigned to the same seucurity zone, L3-Untrusted, as ethernet1/1 had been assigned to.

[AndrewMoore]

Thanks @reaper.  We have a Cisco 3400 fiber switch that  is our next hop from the 3020. Within the static-route tab of the default-vr I did not change the next hop.

Since the fiber switch remains as our next hop I don't believe anything is out of place with regard to  next hop configuration.

Appreciate the suggestions about clearing arp cache and checking arp stats!

[AndrewMoore]

This was the state of our fiber port yesterday before I abandoned ship and moved back over to copper on eth1/1

bob@pafw> show interface ethernet1/13
--------------------------------------------------------------------------------
Name: ethernet1/13, ID: 28
Link status:
Runtime link speed/duplex/state: 1000/full/up
Configured link speed/duplex/state: 1000/auto/auto
MAC address:
Port MAC address d4:f4:be:ab:cd:ef
Operation mode: layer3
Untagged sub-interface support: no
--------------------------------------------------------------------------------
Name: ethernet1/13, ID: 28
Operation mode: layer3
Virtual router default-vr
Interface MTU 1500
Interface IP address: 17.22.113.34/28
17.22.113.37/32
Interface management profile: N/A
Service configured: IKE
Zone: L3-Untrusted, virtual system: vsys1
Adjust TCP MSS: no
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Physical port counters read from MAC:
--------------------------------------------------------------------------------
rx-broadcast 0
rx-bytes 0
rx-multicast 0
rx-unicast 0
tx-broadcast 8
tx-bytes 512
tx-multicast 0
tx-unicast 0
--------------------------------------------------------------------------------
Hardware interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 1614
bytes transmitted 336
packets received 13
packets transmitted 8
receive incoming errors 0
receive discarded 0
receive errors 0
packets dropped 0
--------------------------------------------------------------------------------
Logical interface counters read from CPU:
--------------------------------------------------------------------------------
bytes received 1614
bytes transmitted 336
packets received 13
packets transmitted 8
receive errors 0
packets dropped 0
packets dropped by flow state check 0
forwarding errors 0
no route 0
arp not found 13
neighbor not found 0
neighbor info pending 0
mac not found 0
packets routed to different zone 0
land attacks 0
ping-of-death attacks 0
teardrop attacks 0
ip spoof attacks 0
mac spoof attacks 0
ICMP fragment 0
layer2 encapsulated packets 0
layer2 decapsulated packets 0
--------------------------------------------------------------------------------

bob@pafw>

[TranceforLife]

Hi Andrew.

 So both ports are in the same subnet? Any logs from the Cisco switch? ARP table 

[AndrewMoore]

Hey @TranceforLife,

Our ISP has assigned us a block of /28 IP addresses. In my description I was trying to emphasize that I copied/mirrored the same settings on eth1/13 from eth1/1. As a result both interfaces have the same addresses assigned, just not at the same time.

In other words,  before pulling the copper line and installing the fiber I did the following to produce a valid configuration that would commit. After the following steps I pull copper and plug in optical.

1) Assign eth1/13 to same Security Zone as eth1/1 - that is L3-Untrusted.

2) Remove IP addresses AAA.BBB.CCC.34/28  &  AAA.BBB.CCC.37 and assign to eth1/13

3) Modify default-vr to include int1/13 and assign as interface to be used for the Default Route.

4) Move IKE Gatway interface assignment from eth1/1 to eth1/13

I do not own, manage, or have console access to the Cisco switch but could inquire with the NOC for any specific statistics if you have any suggestions.

Much appreciated.

[TranceforLife]

Hi Andrew,

Try to run PCAP on the 1/13 interface to see for the ARP packets. Configure the filter for this interface and capture Receive and Transmit packets only.  KB article below:

https://live.paloaltonetworks.com/t5/Learning-Articles/How-to-Run-a-Packet-Capture/ta-p/62390

Try GUI option, it is easier 

Cheers

[Roby_Sreejith]

Have you verified the NAT policy?

Just make sure you have migrated everything belongs to e1/1 to e1/13 in NAT

[AndrewMoore]

Excellent suggestion @Roby_Sreejith!

In my previous posts it's clear that I did not modify the Dynamic NAT policy to use eth1/13 instead of eth1/1. 

However, I was trying to ping 8.8.8.8 from the Palo Alto CLI using the AAA.BBB.CCC.34/28 and AAA.BBB.CCC.37 IP addresses, which, I suspect, do not get translated using NAT.

Hope my assumptions are right but happy to be wrong and learn something new (-:

I'll be switching over again today around 12:15 CST so I'll be sure to try your and other suggestions, like clearing ARP tables, capturing packets and filtering ARP traffic, etc.

Thank you again!

[TranceforLife]

Agree with Roby.

Please confirm Cisco port MAC address and that you actually can see it in your arp table:

> show arp ethernet1/13

Also, when pinging 8.8.8.8 try to source the ping from the fibre interface (ethernet1/13)

> ping source (fibre interface IP) host 8.8.8.8

Cheers

[AndrewMoore]

Thanks again @TranceforLife. I can confirm that during the switch to fiber on ethernet1/13 I ran pings from the PA's eth1/13 interface like so:

> ping source AAA.BBB.CCC.34 host 8.8.8.8

I will confirm the MAC address tied to the current interface (Cisco) connected to eth1/1. 

Thank you to everyone for the big assist this morning.

[AndrewMoore]

Thank you to everyone for your suggestions. 

Wanted to let you know the conversion from copper to fiber was a success over lunch today

The solution was I had to issue gratuitous ARPs from a few IPs in my address block - this quickly solved everything.

Cheers to everyone!

[TranceforLife]

Hi Andrew,

Glad to hear it! You can mark this topic as "fixed" now

All the best.