Devices pinging only after clearing ARP table

[BishnuA]

I am having an strange issue PA firewalls reachability to some of the nodes on the network.

I have 10 identical devices connected behind this access switch and all 10 devices are accessible from the access and the core switch at any time. However, only 6 of them are reachable from the firewall at any time.  The remaining 4 devices only pings when ARP is cleared in PA firewalls. The same devices does not ping if pinged from the firewall after 5 minutes. The issues been so random that some of the remaining 4 devices ping randomly sometime of the day.

I checked for the IP duplication, and there are no devices that has the duplicate IPs configured. No NAT or ARP Proxy has been configured on the firewall. I also checked MAC address on the switches and ARP table on the firewalls the table still shows the correct entries. 

For testing, I reduced the ARP-Cache-Timeout to 300 seconds (same as MAC-Aging) and all 10 devices are now reachable.

Has anyone come across similar situation and if reducing the ARP timeout on the firewall is an acceptable solution? Any configuration parameters to check other than ARP timeout to resolve the issue?

[Raido_Rattameister]

Instead of reducing arp cache timeout to have shorter outages better figure out what is wrong to have no outages.

Take packet capture of receive state and filter out non-ip traffic only (to reduce what you capture).

This gives you all received arp packets.

If some system stops responding then check firewall arp table what mac address failed IP matches to.

Clear firewall arp cache.

Try to ping.

Check arp cache again - do you see different mac address for that IP now?

And then stop capture and analyze in Wireshark if there is any incoming arp packet that might have caused arp issue.