Episode Transcript:
Hello everyone and welcome to the very first episode of PANCast. Today we are going to discuss upgrading your firewall. This is a pretty common task but if it is not planned for well, it can cause a lot of pain if there are issues.
Now, we don’t have enough time to cover everything in detail but we will go through the high level requirements and if you go to live.paloaltonetworks.com you can get not only the transcript but also some links to additional detailed articles.
So there are four key takeaways for this episode and they are; plan, prepare, execute and validate.
John Arena is a Professional Services Consultant with a background in Technical Support for Palo Alto Networks and a passion for educating and sharing knowledge with customers.
Let’s start with “plan." It is important to understand why you are upgrading and what are the potential impacts. There can be a number of reasons you are planning to upgrade. It could be a new maintenance release to get a fix or to a new major release to be able to use new features. But no matter the reasons, there are a couple of things you should check well before you get to the actual upgrade. The good thing is, all of this info can be found in one place, the release notes.
Even with an upgrade for a maintenance release, it is still good to review the release notes just to see what other issues have been fixed. You can also check if there are any known issues with the maintenance release you are planning to upgrade to. Should be pretty quick and easy right.
Upgrading to a major release however needs a few more checks as this is obviously a more significant change. So aside from reviewing the known issues, there are some other things you need to check on. First is to check for any changes in default behavior. Each major release will have a number of new features added and these can mean some pretty significant changes under the hood. There can be times where the way a particular feature used to work on say 9.1 is not the same as when you upgrade to 10.1. Fortunately this is all documented so there should be no surprises.
The next thing to check are the upgrade and downgrade considerations which again is well documented. The upgrade considerations are important, as along with the changes in default behavior, they make sure there aren’t any surprises when you do upgrade. It is however also worth checking out the downgrade considerations. For the same reasons mentioned that there can be some pretty significant changes when upgrading major releases. Having to downgrade may either require some additional steps or at a very minimum there may be some things you need to be aware of when going back to the previous major release.
And the final part of planning is checking the support matrix. We have a lot of products that work together at Palo Alto Networks like GlobalProtect clients, userID agents and various plugins. If you are upgrading to a new major release and use other products, just check they are all supported together with the new version you will be on.
So you’re now set with the “plan” stage. You understand why you are upgrading and have the specifics on changes and considerations. On to the next step. Prepare.
Now this one is generally always done but sometimes steps can be missed and we’ll talk about a few of them. Along with the planning stage, the “prepare” stage is to make sure the upgrade goes as smoothly as possible but also so you are ready for the unexpected. Let’s face it, we have probably all been in that situation. You're doing an upgrade late Saturday night as this is the only time these firewalls can take an outage. You’ll be done in an hour and can then call it a night. Pretty simple. Download, install, reboot and away we go. And then halfway through you get the call from the help desk. Email is not working. A few minutes later, Internet access is down as well. Here we go.
So what are a few things that can help? Firstly if you have done the planning stage you should be in a much better position already. You now know that as part of this upgrade there are some SAML changes required on the IdP side. You have already contacted your SAML provider and they are ready to make the changes during your change window. Otherwise what happens? You raise a critical case with Palo Alto Networks TAC because authentication is broken. The engineer starts looking at the issue and the relevant info is found on the upgrade considerations. But because the changes are needed on the SAML provider end as well you need to decide whether to back out the upgrade or raise another critical case with the provider. It is worth the 10 minutes to review the upgrade considerations right?
The other things are really about helping troubleshoot in case of issues and making sure the upgrade is successful. So add some things to your upgrade run sheet like taking config backups, take a tech support file before and after so you have that snapshot. If it’s a critical environment then spend a bit longer on this. Take some detailed snapshots like full routing tables, peer status, interface status and arp table.
Make sure you also include time for testing applications in the change schedule and just as important, factor in that backout time. Hopefully it won’t be required but better to have it in there just in case.
So we have now planned and we have prepared and if you invest the time in these two steps then the next two should be relatively straight forward. Time to execute.
We are now ready to do the upgrades. As I said having already planned and prepared we should be in a pretty good position now but that’s not to say there will be no risks. There can always be some unknowns but having done the previous two steps means there is less chance of surprises.
The key to the “execute” step is to follow the detailed plan you have put together from the planning and preparation stages. Don’t take shortcuts. Hopefully you will have an idea of all the pre and post steps along with the actual upgrade process and all going well should be as per your plan.
I want to talk a bit about the actual upgrade process though and some things to look out for. The main thing is that with any upgrade, a device is being rebooted and therefore there will be environmental changes within your network. This is even more important in a high availability setup. So things like switch ports going down, ARP tables updating and routing changes can all be expected. That’s why as part of the planning stage we make sure we add collecting some state information before the upgrade. It can help us check what things were like if there are issues after the upgrade or even after a failover if upgrading in a HA environment. Also remember that it may be possible in a HA environment there has not been a failover in years. We may not even know if failover works correctly. You can always plan to do the failover test before any upgrades just to make sure. Another benefit with high availability is you can upgrade one firewall first and test to make sure everything is ok.
We now come to the last part. Validate. The validate part can vary depending on the devices you are upgrading so there’s no one size fits all test plan for this. Datacentre firewalls may require a lot more testing, including full application testing. For a smaller branch office firewall it may be ok with just firewall and network checks.
There are some things you can do as the administrator of the firewall to make sure there are no surprises come Monday morning.
The dashboard is a great place to start so you can see the overall health of the firewall and make sure there are no warning signs. At a high level you can also check the system logs for anything that does not look right. You may be seeing session and traffic logs that appear ok but maybe there is a problem with an external service not connecting. For example your LDAP servers for group mapping. It may not be obvious now but when users start trying to access services it could cause real problems.
If the system logs look ok then a quick check of the session table and traffic logs can also help. Do the traffic logs look ok? Are the session end reasons what you would expect or are all sessions showing as aged-out? It shouldn’t take too long to review but can help in confirming the environment is working as expected. Better that any issues are known about now rather than later when users are affected.
If it is a critical setup then also collect the post change details, like tech support file, etc. Compare some of the captures to make sure it looks the same. As an example, compare the pre and post route tables. All these checks can help avoid a major incident should there be any issues that arise from the upgrade.
And there we have it. All going well the upgrade should be a success. Obviously we can’t guarantee there will be no unforeseen issues but by following the plan, prepare, execute and validate process you will certainly be in a much better position to ensure a successful upgrade.
I really hope this has been a helpful overview. There are a lot of detailed articles on upgrading which get down to the nitty gritty so just search for PANCast on LIVECommunity which is at live.paloaltonetworks.com and you can find the transcript of this episode along with links to the relevant detailed articles.
So remember: Plan, prepare, execute and validate. These are your four key takeaways for upgrading your firewall successfully.
Check out the full YouTube playlist: PANCast: Insights for Your Cybersecurity Journey.
Related Articles:
