Ok, as promised we did do some firewall upgrades this week and here are our thoughts.
First we updated one of our remote sites from 7.1.10 to 8.0.7 (PA-3020). We did this by upgrading to 8.0.0 then rebooting and upgrading to 8.0.7. We have not found any problems with. Both before and after Panorama pushed without problems.
This site has: MPLS, IPSEC VPNs, LDAP, Security/NAT/PBF rules, virtual and physical interfaces.
This site does not have Global Protect configured or HA.
We later upgraded our passive HA firewall to 8.0.7 (PA-3020). As this was a secondary firewall we upgraded directly to 8.0.7 (not recommended for some firewall versions but the PA-3000 series was not in that list so we tested it). It upgraded fine and we were able to push configs from Panorama after the upgrade without problems. After confirming everything was still present and correct we failed over from our Primary HA 7.1.10 to the Secondary HA 8.0.7 without problems.
This site has: MPLS, IPSEC VPNs, LDAP, Security/NAT/PBF rules, HA (active/passive), GP Portal/Gateway, virtual and physical interfaces.
We learned two things but only one of them I think will apply to anyone else.
There is a setting that specifically relates to AD users prefix while using LDAP lookups (we think this only affected GP connections but are not 100% sure).
Device -> Authentication Profile -> <Auth_Profile> -> User Domain = <domain>
In this field we had previously entered the DOMAIN.local the new value must just be DOMAIN in the sense that you would type DOMAIN\USER. The problem we found is that in 7.1 (and older) the user would authenticate and the DOMAIN would be added, no problems here everything was happy. After upgrading to 8.0 we found that the system had now changed to take the literal prefix and was now showing users as DOMAIN.local\USER. They would still connet through GP however every rule in the firewall that had a user based restriction/requirement would now reject these users. DOMAIN.local\USER was not valid anywhere.
The second thing has to do with an odd internally built HA active/passive system we are using that we had to shoe horn into doing what we want. The bottom line "I think" is that when we failed over from Primary to Secondary and did not fail back this system lost connectivity. Not because the failover caused the session to become invalid but "we think" the system is doing more than checking session state. I don't have details yet but "I think" it is doing not only a session state check but also a physical check on the path it is taking between active/passive devices. It is checking something on the physical side (MAC or something else) and if this changes the "we think" the session is no longer considered valid. Where this became a problem is that historically we have done reboots and Primary failed to Secondary then after the reboot Secondary failed to Primary as Secondary was then updated and rebooted in turn. This process meant that the Secondary firewall was only the active one for 5-6 minutes. We believe the failover state of the system we are working with is closer to 10 minutes so when we made this change and did not fail back over to Primary the system did not see a recovery of the session in time (no hardware match??) and failed over to the passive remote system. "We think" the system either did not create a new session as the old session was invalid or the time out for creating a new session does not match the failover timeout.
We will clearly need to do more testing. As a network engineer I don't really have any involvement in this system (till it breaks, lol) or in how it was built. I may or may not have an update on this, I'm not sure if it matters to other organizations as this was internally built.
I sucessfully upgraded over the weekend. Only issue I had was that I updated one all the way up to 8.0.7 (kept suspending it before it became active after every reboot to minimize traffic interruptions) and, when it tried to come online, saw the large version difference between it and the already active peer. I ended up suspending the active peer and the upgrade box came online. This was going to happen anyways so not a big deal.
So far I did not see any issues with the VPN configs.
Wildfire started going crazy after the upgrade and enabled me to immediately detect a problem on our network. I found this interesting since, presumably, the previous software was using the same Wildfire content so, unless an update came out during my upgrade window, it seems like some improvements were made to the Wildfire engine that allowed it to detect something it couldn't previously.
Overall it went really well and pretty much to plan. Naturally, 8.0.8 released within a week after and includes some fixes for A/A setups so I'll have to watch our 8.0.7 deployments to see if they encounter any issues until 8.0.8 becomes recommended.
Thanks again everyone for the responses!
Glad to hear it went well for you @jsalmans.
Yeah, we were warned about them not being able to sync more than one major revision apart. Did you notice any session drops when you forced the fail over from 8.0 to 7.0?
I didn't get any complaints but I was doing it between 6 am and 8 am on a Saturday morning which is a low traffic time for us.
I expect there was since one firewall was told to stop responding and the other was not already operating as an active. Generally I see some traffic drops when I put one in standby due to a combination of OSPF convergence and the really unfortunate way I'm having to do traffic routing via Policy-Based Routing and ping watchdogs (I keep running into issues with "set ip next-hop recursive" and the hardware I'm using).
Click Accept as Solution to acknowledge that the answer to your question has been provided.
The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!
These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!
The LIVEcommunity thanks you for your participation!