Incomplete ARP Entry or Firewall Responds to Every ARP Request on the Network

There can be two symptoms:

The Palo Alto Networks firewall has an incomplete ARP entry for a host on the network (for example, default gateway):

> show arp all
maximum of entries supported :      2500
default timeout:                    1800 seconds
total ARP entries in table :        1
total ARP entries shown :           1
status: s - static, c - complete, e - expiring, i - incomplete

interface         ip address      hw address        port              status   ttl
--------------------------------------------------------------------------------
ethernet1/4       10.108.121.1    (incomplete)      ethernet1/4         i      1

The firewall is responding to every ARP request on the network. On the endpoint, select any random IP address, try to ping it and you'll see an ARP entry with the firewall's IP MAC:

$ ping -w 2  10.108.121.251
$ ping -w 2  10.108.121.252
$ arp -an
? (10.108.121.251) at 00:1b:17:00:04:13 [ether] on eth4
? (10.108.121.252) at 00:1b:17:00:04:13 [ether] on eth4

With a large number of NAT rules, it can be difficult to narrow down the policy. Three methods to identify the NAT rule are described below. The first two are safe to perform, the third option involves enabling debugs on the dataplane and should be used cautiously.

Method 1

Identify the offending pool:

> show running global-ippool
Idx  Type             From                             To              Num   Ref. Cnt
---- ---------------- -------------------------------- --------------- ---   ----------
3    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.5    1     1
6    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.0    256   1

Note: Pool 6 is using 256 addresses in 10.108.121.0 network.

To determine which policy, run the following command, then press "/" (slash), then type in: "idx: 6" (there is space between double colon and 6 and if needed go back a little bit by pressing the up arrow key).

> show running nat-policy
[...]
dmz_Out {
        from dmz;
        source any;
        to outside;
        to-interface ethernet1/4 ;
        destination any;
        service  any/any/any;
        translate-to "src: 10.108.121.0-10.108.121.255 (dynamic-ip-and-port) (pool idx: 6)";
        terminal no;
}

See the incorrectly configured rule is dmz_out.


Method 2

Run a single command, which basically tells the firewall to output all rule names and src NAT translations, where a range of IPs is used. In this case, the rule name that precedes the translation is the offending rule.

> show running nat-policy | match {\|src:[^\(]*-

"Rule 1" {
smtp04-in {
smtp04-out {
smtp03-out {
"Internet outbound" {
dmz_Out {
        translate-to "src: 10.108.121.0-10.108.121.255 (dynamic-ip-and-port) (pool idx: 6)";
"Rule 4" {
"Rule 5" {
smtp03-in {

Method 3

Important! Use cautiously, because this method enables debugs on the dataplane.

Enable debug on DP:

> debug dataplane packet-diag clear all
> debug dataplane packet-diag set filter match non-ip only
> debug dataplane packet-diag set filter on
> debug dataplane packet-diag set log feature flow arp
> debug dataplane packet-diag set log on
> debug dataplane packet-diag clear log log

After trying to send the communication through the firewall (or pinging from the firewall default gateway):

> ping source 10.108.121.253 host 10.108.121.1

Review the DP debug files:

> less dp-log pan_task_*

It is possible to go to the next file by pressing "n"

At some point, the following appears, which may be similar to:

Received ARP packet from port ethernet1/4
Packet decoded dump:
L2:     00:50:56:a3:10:5a->00:1b:17:00:04:13, type 0x0806
ARP:    hardware type 0x0001
        protocol type 0x0800
        hardware size 6
        protocol size 4
        opcode REPLY
        sender mac address 00:50:56:a3:10:5a
        sender ip address 10.108.121.1
        target mac address 00:1b:17:00:04:13
        target ip address 10.108.121.253
ARP packet sent from translated IP in NAT rule index 5 in vsys 1
ARP packet sent to interface ethernet1/4 IP
ARP packet parse complete, learn: no, target myself: yes, gratuitous ARP: no

In the example above, the firewall states that someone is using the IP address, which firewall it is using in NAT rule index 5.

Note: Index 5 means only active policies (disabled policies do not count) and it starts from 0. The easiest way is to again run command "> show running nat-policy" and count policies.


Fix Details

This is how the rule looked in the WebGUI and CLI:

> show config running
[...]
dmz_Out {
     to outside;
     from dmz;
     source any;
     destination any;
     service any;
     nat-type ipv4;
     source-translation {
          dynamic-ip-and-port {
               translated-address 10.108.121.211;
          }
     }
     to-interface ethernet1/4;
}

At a glance, there is nothing wrong with this rule, however after some investigation see that the 10.108.121.211 is really an object:

10.108.121.211 {
              ip-netmask 10.108.121.211/24;
            }

This would be difficult to find just by browsing through the WebUI.

Change the netmask to the appropriate one (most likely /32) and verify. See how the rule looks after the change:

> show running nat-policy
dmz_Out {
        from dmz;
        source any;
        to outside;
        to-interface ethernet1/4 ;
        destination any;
        service  any/any/any;
        translate-to "src: 10.108.121.211 (dynamic-ip-and-port) (pool idx: 7)";
        terminal no;
}

> show running global-ippool
Idx  Type             From                             To              Num   Ref. Cnt
---- ---------------- -------------------------------- --------------- ---   ----------
3    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.5    1     1
7    Dynamic IP/Port  0.0.0.0-255.255.255.255          10.108.121.211  1     1

In the example above, note a different pool ID used by the very same rule, but only a single IP address is used.