Issues and Observations Upgrading to PAN-OS 7.1.7

[Scott_Sadlocha]

Hello Everyone,

We are in the process of upgrading our firewalls to PAN-OS 7.1.7, and have come across a number of issues and oddities, and figured
it would be a good idea to create a community post to document them so that they might be a reference for others. Some items to note
follow.

We don't have a large firewall environment, only about 10 devices total. Most of our devices are 3000 series devices, with the
exception of one smaller PA-500. We are upgrading from 7.0.5-h2 to 7.1.7.

1. The first issue had to do with the download of the pertinent files. On every firewall, I had to download two files twice due to a
strange issue. I would download 7.1.0 to the firewall and it would complete successfully. I would then download 7.1.7, and it would
fail, indicating that the base file (7.1.0) needed to be downloaded first. I would then refresh the Software screen, and all of a
sudden, 7.1.0 would be gone. I would then go through the process again, downloading 7.1.0 and then 7.1.7, and it would work the
second time. No other admins were working on the firewalls or anything else that might overwrite my work. I had to do this on every
firewall and the behavior was the same.

2. The second issue has to do with a firewalls terminating either end of a DR tunnel. We have an ipsec tunnel to an offsite location.
The offsite location has 2 firewalls, one of which has been upgraded successfully (it does not terminate the tunnel). If I upgrade
the firewall on the far end of the tunnel, both of those firewalls stop communicating to Panorama. If I upgrade the firewall on the
near end of the tunnel, it communicates to Panorama, but both of the firewalls at the offsite location stop communicating to
Panorama. I have tried this several times, and end up having to downgrade each time. All other traffic traverses the tunnel! I can
log into the firewalls at the remote location, and all traffic is fine. The logs even show the traffic passing, with no blocks.
However, they show Disconnected in Panorama. I currently have a support case open on this issue, but so far, no luck.

3. The third issue had to do with Wildfire updates. As soon as the upgrade finishes on a Wildfire enabled firewall, Wildfire updates
started failing. They would download, but the install would fail. I could go to the PA support site, download the file to my
computer, and then install from file on the firewall, and Wildfire would update. However, any subsequent Wildfire content updates
would only download but not install. After troubleshooting and then finding an old forum post that indicated I might have to reinstall the OS, I opened a ticket with support. I got some tips and tried deleting content updates via CLI and several other things, but had no luck. Eventually, another community member posted that they had this issue and resolved it by changing the Update Server info. I changed our update server from staticupdates.paloaltonetworks.com to updates.paloaltonetworks.com and it resolved the issue for us! I would never guess something so simple would be the fix. Since the files were downloading and not installing, I had concentrated my efforts on the firewalls and not the download location. It seems that staticupdates.paloaltonetworks.com is serving up incorrect files or it may be something else I am not aware of.

4. The fourth issue had to do with traffic blocks. Immediately after upgrade, we had mssql traffic getting blocked, even though the traffic did not change, and we had made no changes to the rules. After talking to support, we found that the way application-default is handled changed with 7.1. While this is documented, it can be a bit tricky. Any rule that uses an application of Any, will now enforce application-default ports. Our mssql rule was like this. The mssql traffic in the application description shows default ports of 1433/1434. However, if your MS SQL environment uses a cluster, those default ports will be different. The traffic will still classify as mssql, but will use ports in the 54000 range. This immediately caused issues with our traffic, though it wasn't showing as blocked in the firewall logs. It was very strange. Once I switched the port to "Any", it immediately started working.

Well, there goes our major issues so far. We have seen some minor issues as well, but those haven't been too bad. These include column and log settings in the GUI not being retained, and a couple others. I have to say, after having no issues on previous upgrades, I may have been spoiled, but this is the most problematic upgrade I have had yet on Palo Alto firewalls. I am hoping to get the remaining two firewalls terminating the ipsec tunnel upgraded at some point, once I figure out what the problem is. In my last go-round, I took tech support package files before and after the upgrade, so I am hoping these offer some clue. I am hoping this info helps someone.

[TranceforLife]

Hi,

Cannot comment much on first 2 strange issues, 

3) For me its always should be FQDNs as below:

4) Expected behaviour as per article below:

https://live.paloaltonetworks.com/t5/PAN-OS-7-1-Articles/PAN-OS-7-1-Policy-behavior-change-applicati...

Thx,

Myky

[BPry]

Same as @TranceforLife; I've never seen anyone use staticupdates.paloaltonetworks.com, I've only ever seen it as update.paloaltonetworks.com or downloads.paloaltonetworks.com. Was this something that support did at one time for troubleshooting?

Again 4 is very much expected behavior and PAN did a pretty good job of trying to get the message across via the release notes, emails sent out on 7.1s release, and a few articles on here about the change. Unfortunately I'm not sure how they would accomplish getting the word out any better. 

[Scott_Sadlocha]

Thanks for the responses, it is greatly appreciated. The update URLs were in place from a previous firewall engineer. I am glad that was such an easy fix.

As to the fourth point, yeah, that was a total miss on my part with regard to the change. But, I still wanted to get the info out there regarding the SQL Cluster behavior, in that the traffic classifies the same, but the ports will be different. Thanks again-