Pushing updates from Panorama to PA220 uses incorrect IP address

[fjwcash]

We have Panorama installed in our DMZ, behind a PA5220.  Management interface for Panorama has a 10.10.10.x IP, which gets NAT'd to a public IP.  Currently running Panorama 8.1.10 in panorama mode.

All of our firewalls (currently PA200/PA500 with PanOS 7.1.x) connect to Panorama using the public IP.  All of the firewalls have private 10.x.x.x IPs for their management interface, and Panorama traffic is NAT'd by the firewall to a public IP.  All connections between the firewalls and Panorama are done using public IPs.

We've been using Panorama successfully for 5+ years to push updates to our firewalls (software upgrades, dynamic updates, policies, configs, etc).  And it still works to push update

We have a brand new PA220, currently running 8.0.7, with the management port plugged into the LAN at one site (so it has a private IP in the same 10.x.x.x subnet as the firewall currently there).  It's configured with the public IP for the Panorama server, and is showing as connected in Panorama.  We've pushed the licenses and dynamic updates to it from Panorama.  However, when trying to push an OS upgrade, the PA220 tries to connect directly to the private IP of the Panorama server!  Which fails, obviously.

All of the port 3978 and 443 traffic to/from the Panorama server is correctly NAT'd.  It's only port 28443 traffic that the firewall tries to connect to the Panorama server's private IP.

Pushing a software update from Panorama to a PA200 running PanOS 7.1.19 works.  All of the traffic goes over the existing session (initiated by the firewall connecting to Panorama on TCP/3978).  There is no traffic whatsoever using TCP/28443.

Pushing a software update from Panorama to a PA220 running PanOS 8.0.4 fails.  Initial connection is done using the existing session on port TCP/3978, but then the firewall initiates a new connection to the private IP of the Panorama server on TCP/28443, which fails.

Has anyone seen anything like this before?  Has something changed with the way Panorama 8.1 works or that PanOS 8.x works when connected to Panorama 8.1?

All of our Panorama traffic has always gone to/from the public IP.  It's very odd to see the private IP showing in the logs on firewalls on the other side of town.  That usually only happens with ancient / brain-dead protocols that embed the server IP inside the data payload of the protocol, instead of using the IP headers.

Just as a test, I did a software push (not install/reboot) of 8.0.0 to a PA200 running 7.1.19 and it succeeded without issues, and the 10.10.10.x IP of the Panorama server never showed anywhere in the logs (just the public IP).

[fjwcash]

Oh, and connecting directly to the web interface on the firewall running 8.0.7 and doing a software upgrade from there succeeds.

It's only when pushing an OS upgrade from Panorama 8.1.10 to the firewall running 8.0.7 that fails because the firewall wants to connect direct to the 10.10.10.x IP of the Panorama server.

[fjwcash · Accepted solution]

Aha!  It *IS* a brain-dead protocol!  It's FTP all over again, but in 2019?

Panorama 8.1 includes a new field in the Panorama tab --> Setup --> Interfaces sub-tab --> Management dialiog:  Public IP Address.  This wasn't there in 6.x (which we started with) and guessing it wasn't there in 7.1 (as we had no issues with it), but is there now in 8.1.

Which means, the paloalto-updates protocol (that's what the application shows up as in the logs now) is sending the Panorama server's IP address in the payload!  This is a serious no-no in this day and age of NAT everywhere!

I am seriously amazed that this made it through QA at Palo Alto.  What's the point of setting the IP address of the Panorama server on the Device tab of the firewall, if the Panorama server is going to tell it to use a different IP?