ARP Proxy didn't work after PanOS upgrade from 10.1.6 to 11.1.2-h3

[Eric_Troldahl]

Short description:  We upgraded to 11.1.2-h3 on an HA pair of PA-820s last night, and the NAT/ARP Proxy that was functioning on PanOS 10.1.6 is no longer functioning.

We would really prefer a simple solution on 11.1.2-h3 rather than going through a 5 hour downgrade that we would still need to upgrade by the November 18th drop dead date of Advisory: https://live.paloaltonetworks.com/t5/customer-advisories/update-to-additional-pan-os-certificate-exp... 

Is there something we can do like adding the Interface IP to the NAT rule, or setting an explicit ARP Proxy instead of just relying on PanOS to figure it out?

One possible complication is that our interface ae1.xxxx has two IP addresses.

TL;DR:

- Short Term we have a workaround by talking the Network team into temporarily adding TWELVE static ARP entries.

- This used to work until last night.

- The only changes we made to the firewall (PA-820) were to upgrade from 10.1.6 to 10.2.6-h3 to 11.0.4-h2 to 11.1.2-h3.

- The upgrades took 5 hours (did I mention PA-820?) so I REALLY don't want to have to downgrade, especially since 10.1.6 doesn't have the Device Certificate patches, so we would need to upgrade to SOMETHING before November.

TIA for any insight!

Eric Troldahl

Firewall Lead, Michigan State University.

[BPry]

@Eric_Troldahl,

Have you reviewed https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClGZCA0 and attempted any of the standard fixes to this issue? Since you don't detail how your NAT policies were configured previously or if you have the secondary addresses on the interfaces directly it's a bit difficult to troubleshoot what could have possibly changed in your upgrade.

I personally just prefer to make routes whenever possible instead of configuring the address on the interface as a secondary address, but that also works perfectly fine. Just gets a bit tedious if working with a lot of addresses.

[Eric_Troldahl · Accepted solution]

The range defined for NAT was directly attached to an interface.  It turns out that the object that was used to define the interface IP on one of the two subnets on the VLAN had a /32 mask, so none of the other IPs on that Subnet were getting ARP resolution.  We currently changed to a hardcoded IP/mask (that can cause a "Ghost"), but we are discussing an architecture naming convention change that would require separate objects for Interface addresses, using a name like "vlan3100-subnet-192-168-1-0-interface" so no one would use it except in a context that needs to change if the IP or VLAN changes. 

NOTE:  We have a very large network with not just RFC1918 private IPs, but also a public /13 range, and some of the RFC 1918 addresses need to NAT to the public range even within the campus to be allowed to route from Server and DMZ zones out to VPN tunnels, local Untrust zones, and even Internet Untrust ranges.