topic Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama? in General Topics

Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Tue, 30 Jan 2018 22:33:45 GMT

Greetings all,

Title pretty much says it all but we're wanting to move to 8.0.6 since it is a Palo Alto support recommended version. We're currently running 7.1.4 I believe with both of our active/active firewalls. Panorama is already on the 8.0.x track.

My normal update procedure is to apply the update to one firewall, let it reboot and come back online and start passing traffic and then do the same to the other one. Usually only results in 5 to 10 seconds of dropped traffic when each firewall goes down.

I figured there is probably no difference in the actual upgrade procedure but I wanted to check to see if anyone has run into any gotchas or any settings that need to be modified once the new version is online that might result in a longer outage otherwise?

Thanks!

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Tue, 30 Jan 2018 23:37:09 GMT

I don't have a lot to add but we looked into this as well as we are A/P with ours HA pair. The firewalls in 7.1.x and 8.0.x will still synchronize after one is upgraded and rebooted, anything older (7.0.x etc) will not synchronize.

Brian

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BPry — Wed, 31 Jan 2018 16:37:25 GMT

@jsalmans,

I would recommend following the recommendation detailed in this document HERE. Also keep in mind that 8.0 does some interesting things with the logs, so that takes a few minutes to get everything working correctly.

Keep in mind that the upgrade procedure best practices recently changed later last year to the following.

Old Method:

Upgrading to a new major version for another with maintenance releases already available was simply to download the base image and the maintenance release, and only install the maintenance release.

New Method:

Upgrade to the latest maintenance release within your current major version; then install the base 8.0.0 and restart before proceeding to 8.0.6.

Why:

The old method worked perfectly fine because the firewall is able to explode the base image and the maintenance image installer packages to pick apart all of the pieces and parts required to form an installer image for a direct upgrade to 8.0.6. However with the larger file sizes of the new releases PA started to see some issues with firewalls with limit storage, primarly the PA-200/220, PA-500, PA-2000, and the PA-4000.

If your not running an effected platform and you know that you have plenty of system space to explode the images you can still use the old upgrade recommendation to keep downtime to a minimal. Just know that you wouldn't be following the best practices as recommended and it can cause issues if you don't actually have the disk space required to perform this action.

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Wed, 31 Jan 2018 18:49:44 GMT

Another note. We upgraded our Panorama to 8.0.7 and did not have any problems. We did this based on a security release that was fixed in 8.0.7 and was recommended by our SEs.
https://www.paloaltonetworks.com/documentation/80/pan-os/pan-os-release-notes/pan-os-8-0-7-addressed-issues#_49938

Another thing we are finding out right now based on a support ticket is that from our current 7.1.10 there is a bug going to 8.0.x that Panorama will blow out the VPN configurations unless your firewalls are on 7.1.15.

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Wed, 31 Jan 2018 19:06:06 GMT

@BrianRa thanks for that information, I'll definitely move to 7.1.15 first.

Out of curiousity, what security bug did they reocmmend moving to 8.0.7 for?

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Wed, 31 Jan 2018 19:18:05 GMT

@BPry thanks for the info. Panorama is already at 8.0.6 and we switched it to the new logging. Does anything have to be done on the firewalls themselves for the logging change or does it do it automatically?

My current plan for upgrading is:

7.1.14(current)->7.1.15->8.0.0->8.0.6

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Wed, 31 Jan 2018 20:17:45 GMT

Based on what support is saying and what we have been reading that install order looks correct. That is what we will be doing next week when we upgrade our first PA-3020 (only to the 8.0.7).

This is the email we received.

PAN-SA-2017-0030 - Cross Site Scripting in PAN-OS GlobalProtect

* Medium Severity

* Fixed in PAN-OS 6.1.19, PAN-OS 7.0.19, PAN-OS 7.1.14, PAN-OS 8.0.6-h3

* Affects PAN-OS GlobalProtect portal and gateway

* CVE-2017-15941

PAN-SA-2017-0031 - Cross Site Scripting in PAN-OS Captive Portal

* Medium Severity

* Fixed in PAN-OS 8.0.7

* Affects PAN-OS captive portal

* CVE-2017-16878

PAN-SA-2017-0032 - ROBOT attack against PAN-OS

* High Severity

* Fixed in 8.0.7

* Affects PAN-OS SSL/Decryption and GlobalProtect portal and gateway

* CVE-2017-17841

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Wed, 31 Jan 2018 20:21:47 GMT

Is 8.0.7 still in review phase? I don't mind upgrading to that to fix security issues but I don't necessarily want to land on a version not currently recommended by Palo Alto after upgrading between major versions.

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Wed, 31 Jan 2018 20:31:42 GMT

It is production, our SEs and support did not have any concerns with it when we have talked to them about it. Panorama has run fine. Remember we haven't done a firewall yet but I haven't been able to find any negatives posted. If we do ours first before you I will update with anything we noticed. This will be a remote site, we didn't want to tackle the VPN possible problems and the major revision update at the same time first as we have this option.

Brian

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BPry — Wed, 31 Jan 2018 20:46:00 GMT

@jsalmans,

Can confirm that most of my firewalls are now running 8.0.7 without any issues.

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Mon, 05 Feb 2018 19:28:55 GMT

Thanks for the info. I think I'll move to 8.0.7 for security since it seems like it might be close to being recommended anyways.

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Sat, 10 Feb 2018 00:39:36 GMT

Ok, as promised we did do some firewall upgrades this week and here are our thoughts.

First we updated one of our remote sites from 7.1.10 to 8.0.7 (PA-3020). We did this by upgrading to 8.0.0 then rebooting and upgrading to 8.0.7. We have not found any problems with. Both before and after Panorama pushed without problems.

This site has: MPLS, IPSEC VPNs, LDAP, Security/NAT/PBF rules, virtual and physical interfaces.

This site does not have Global Protect configured or HA.

We later upgraded our passive HA firewall to 8.0.7 (PA-3020). As this was a secondary firewall we upgraded directly to 8.0.7 (not recommended for some firewall versions but the PA-3000 series was not in that list so we tested it). It upgraded fine and we were able to push configs from Panorama after the upgrade without problems. After confirming everything was still present and correct we failed over from our Primary HA 7.1.10 to the Secondary HA 8.0.7 without problems.

This site has: MPLS, IPSEC VPNs, LDAP, Security/NAT/PBF rules, HA (active/passive), GP Portal/Gateway, virtual and physical interfaces.

We learned two things but only one of them I think will apply to anyone else.

There is a setting that specifically relates to AD users prefix while using LDAP lookups (we think this only affected GP connections but are not 100% sure).

Device -> Authentication Profile -> <Auth_Profile> -> User Domain = <domain>

In this field we had previously entered the DOMAIN.local the new value must just be DOMAIN in the sense that you would type DOMAIN\USER. The problem we found is that in 7.1 (and older) the user would authenticate and the DOMAIN would be added, no problems here everything was happy. After upgrading to 8.0 we found that the system had now changed to take the literal prefix and was now showing users as DOMAIN.local\USER. They would still connet through GP however every rule in the firewall that had a user based restriction/requirement would now reject these users. DOMAIN.local\USER was not valid anywhere.

Brian

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Sat, 10 Feb 2018 00:42:55 GMT

The second thing has to do with an odd internally built HA active/passive system we are using that we had to shoe horn into doing what we want. The bottom line "I think" is that when we failed over from Primary to Secondary and did not fail back this system lost connectivity. Not because the failover caused the session to become invalid but "we think" the system is doing more than checking session state. I don't have details yet but "I think" it is doing not only a session state check but also a physical check on the path it is taking between active/passive devices. It is checking something on the physical side (MAC or something else) and if this changes the "we think" the session is no longer considered valid. Where this became a problem is that historically we have done reboots and Primary failed to Secondary then after the reboot Secondary failed to Primary as Secondary was then updated and rebooted in turn. This process meant that the Secondary firewall was only the active one for 5-6 minutes. We believe the failover state of the system we are working with is closer to 10 minutes so when we made this change and did not fail back over to Primary the system did not see a recovery of the session in time (no hardware match??) and failed over to the passive remote system. "We think" the system either did not create a new session as the old session was invalid or the time out for creating a new session does not match the failover timeout.

We will clearly need to do more testing. As a network engineer I don't really have any involvement in this system (till it breaks, lol) or in how it was built. I may or may not have an update on this, I'm not sure if it matters to other organizations as this was internally built.

Brian

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Fri, 16 Feb 2018 15:17:28 GMT

I sucessfully upgraded over the weekend. Only issue I had was that I updated one all the way up to 8.0.7 (kept suspending it before it became active after every reboot to minimize traffic interruptions) and, when it tried to come online, saw the large version difference between it and the already active peer. I ended up suspending the active peer and the upgrade box came online. This was going to happen anyways so not a big deal.

So far I did not see any issues with the VPN configs.

Wildfire started going crazy after the upgrade and enabled me to immediately detect a problem on our network. I found this interesting since, presumably, the previous software was using the same Wildfire content so, unless an update came out during my upgrade window, it seems like some improvements were made to the Wildfire engine that allowed it to detect something it couldn't previously.

Overall it went really well and pretty much to plan. Naturally, 8.0.8 released within a week after and includes some fixes for A/A setups so I'll have to watch our 8.0.7 deployments to see if they encounter any issues until 8.0.8 becomes recommended.

Thanks again everyone for the responses!

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

BrianRa — Fri, 16 Feb 2018 22:54:33 GMT

Glad to hear it went well for you @jsalmans.

Yeah, we were warned about them not being able to sync more than one major revision apart. Did you notice any session drops when you forced the fail over from 8.0 to 7.0?

Brian

Re: Any special instructions to move A/A firewalls from 7.1.x to 8.0.6 via Panorama?

jsalmans — Fri, 16 Feb 2018 22:59:35 GMT

I didn't get any complaints but I was doing it between 6 am and 8 am on a Saturday morning which is a low traffic time for us.

I expect there was since one firewall was told to stop responding and the other was not already operating as an active. Generally I see some traffic drops when I put one in standby due to a combination of OSPF convergence and the really unfortunate way I'm having to do traffic routing via Policy-Based Routing and ping watchdogs (I keep running into issues with "set ip next-hop recursive" and the hardware I'm using).