topic Re: Quick Note on 8.1.0 Deployments in General Topics

Quick Note on 8.1.0 Deployments

BPry — Tue, 13 Mar 2018 19:43:19 GMT

Since its release we've seen an uptick in folks deploying 8.1.0 to their firewalls, and that's a great thing. I just want to throw out a word of caution before doing so however; while 8.1.0 is one of the most stable base releases Palo Alto Networks has published, you need to do your homework before deploying this in any environment.

LAB Devices:

If you have access to any sort of LAB equipment, this is where you should be installing 8.1.0. Start testing your configuration in a LAB environment so that you can have a knowledgeable estimate of when you feel comfortable deploying 8.1 to your production equipment.

If you happen to utilize your LAB equipment in a Change Management process, take note that you are running a different version of PAN-OS when you actually test changes. Something that didn't work in your 8.1.0 LAB may work perfectly fine on 8.0.8 that you have running on your production equipment. On the other hand, something that works out perfectly fine on 8.1.0, may not function on 8.0.8 due to a bug being patched between versions.

Production Devices:

If you do not have access to LAB equipment to verify that your production configuration will actually fully function on 8.1.0, I would personally highly advise you to keep 8.1.0 off your production equipment.

Limitations of 8.1.0 are fairly small, however there are 13 pages of known issues within 8.1.0 along with 3 known issues specific to a WF-500 appliance. Before you contend with loading 8.1.0 on production equipment you should take the time to go through all of these known issues and decide if your environment would actually experience them and if you can work around them until they are patched in future maintenance releases. Causing an outage because you want to utilize the awesome SSL Decryption Broker, or the awesome new hit counters, is likely not going to go well.

Generally this boils down to following Palo Alto's recommended upgrade procedure and just doing your own due diligence before upgrading to 8.1. I think there are a few people that are getting wrapped up in the truly amazing feature improvements of 8.1, and throwing best practices out the window. If you don't have LAB equipment to properly test things out, let those of us that do find all of the bugs before causing an outage due to wanting a new software upgrade quickly.

If you truly want 8.1 and just simply can't wait to upgrade, I'd at least make a post here about what your configuration looks like prior to upgrading. We have a lot of people within these forums that have been running 8.1.0 since the beta was released on LAB equipment and home deployments that can likely take a glance at what you are doing and at least give you some real-world experience on what you should expect.

Re: Quick Note on 8.1.0 Deployments

CaviumKeith — Tue, 13 Mar 2018 19:41:23 GMT

Great Advice of course. Management at my company is chomping at the bit for a more secure Linux deployment of Global Protect. My test device has it working well, removing the need for the X-Auth PSK and implementing a Public Certificate authentication mechanism was key. Unfortunately, that part isn't supported on the pre 8.1 OS as "Linux" isn't a valid OS option on the Portal Config.

PS, use spell check! Some people in management see misspelling and the author's credibility is instantly diminished regardless of the years of experience.

Re: Quick Note on 8.1.0 Deployments

BPry — Tue, 13 Mar 2018 19:45:33 GMT

@CaviumKeith,

I really wish spellcheck on Live was automatic like most other message boards. I've sent the original post through Word, so hopefully the spelling is at least somewhat correct. Honestly though, I don't think many management personnel are visiting the Live forums.

Re: Quick Note on 8.1.0 Deployments

mlinsemier — Wed, 14 Mar 2018 19:39:04 GMT

I've been playing with PAN-OS 8.1 on a PA-200 and a PA-220 of which there is a site-to-site VPN tunnel between them. The upgrade went well overall (from 8.0.8 to 8.1.0) however I have run into two things, one more troubling than the other:

LDAP - After the update to 8.1 my the LDAP attribute is required and if empty LDAP authentication will fail. In our case I needed to add sAMAccountName to complete one phase of authentication for my Global Protect clients.
Site-to-Site VPN - After the update to 8.1, traffic accross the IPSec site-to-site VPN is sluggish and simple functions such as logging into Active Directory no longer work as it did before. I have even went as far as to create a special rule to disable server response inspection for SMB traffic yet no dice. Other protocols such as HTTPS, RDP, SSH all seem to run fine yet Microsoft workstations have issues logging into the domain. Overall it seems to be a bit slower than before as well.

[Update] As of today they can read from shares but cannot write to them. The intersting thing is that this seems to only affect Microsoft SMB shares via the domain controller. SMB shares on other devices (such as QNAP which I think uses Samba) work without issue. Time to open a support case.

There are the two issues that I have expereince so far. The VPN issue is troubling and I may have to revert to 8.0.8 if i cant figure this one out. If anyone has any ideas, I would gladly listen to them.

-Matt

Re: Quick Note on 8.1.0 Deployments

evanm01 — Fri, 16 Mar 2018 00:30:31 GMT

Did you get any specific details from support on the SMB issue? Perhaps a way to work around it without downgrade?

Re: Quick Note on 8.1.0 Deployments

mlinsemier — Fri, 16 Mar 2018 14:15:04 GMT

After talking to support there were several other cases open against the same SMB issues. Myself and others tried many workarounds, including application override, which did not to resolve the issue. I ended up downgrading to PAN OS 8.0.8 and the issue was resolved.

While I know this is the initial release of a new version, it seems that a bug in something as widely used as SMB would be caught early on in internal and beta testing. Most people using 8.1 as an edge firewall (where SMB is not used) or are not using a VPN would probably have no issues with this release. I still feel like Palo Alto should pull this, it's a pretty big issue in my opinion.

- Matt

Re: Quick Note on 8.1.0 Deployments

BPry — Fri, 16 Mar 2018 14:20:42 GMT

@mlinsemier,

It's a noted issue and those that discuss upgrading to 8.1.0 should be alerted to this issue if the SE is decent. This actually has been kicking around in the Beta forums for a while now, so it was a known issue that for some reason doesn't appear to have been documented in the release documentation.

It's important to note here that Palo Alto Networks isn't recommending people actually upgrade to 8.1.0; that's an important aspect that I think people need to be more mindful of. It is currently not a recommended release.

Re: Quick Note on 8.1.0 Deployments

evanm01 — Fri, 16 Mar 2018 14:22:48 GMT

The fact that this isn't in the Known Issues is a huge problem, especially if it was brought up in beta discussions, "recommended release" or not.

Re: Quick Note on 8.1.0 Deployments

mlinsemier — Fri, 16 Mar 2018 14:59:03 GMT

@BPry

I 100% agree with what you are saying that customers should engage their SE and be cautious with brand new releases, but on the flip side Palo Alto themselves are equally responsible for media blasting "NEW PAN-OS 8.1... GET IT NOW!" and "LOOK AT THESE NEW FEATURES" in which I there aren't a bunch of caution signs displayed. It's almost like Palo Alto is doing the "LOOK AT US, WE ARE BETTER THEN OUR COMPETITION" but then in small fine print it says "... but really don't install this until 8.1.3". Don't get me wrong, I love our Palo Alto products and woulnd't recommend anything else, but maybe they should adopt an early development title or something.

Re: Quick Note on 8.1.0 Deployments

BPry — Fri, 16 Mar 2018 15:03:00 GMT

@mlinsemier,

Yup. Marketing and Sales are pretty heavy hitting departments in any company though, and they generally like to push NEW.

Re: Quick Note on 8.1.0 Deployments

Aiace — Sun, 18 Mar 2018 15:01:38 GMT

Hi,

I agree. I have the same issues after upgrading to 8.1.0. Then I have to downgrade too. Thanks.

Re: Quick Note on 8.1.0 Deployments

teoder — Wed, 21 Mar 2018 16:03:53 GMT

Had the same issues with 8.1.0 and went back to 8.0.8.

We experienced slow/non-working domain logons and SMB/CIFS/DFS Shares.

If I ran "show session all filter state discard application ms-ds-smbv3" I had lots of sessions discarded. If I looked in detail on one of those sessions with show session id [session-id] I could see that they were discarded due to "resources-unavailable".

Some users report that 'Application Override' might be a way forward until the issue is fixed.

https://live.paloaltonetworks.com/t5/General-Topics/PAN-OS-8-1-0-SMB-Issues/m-p/205760

Re: Quick Note on 8.1.0 Deployments

ITSysEng — Thu, 19 Apr 2018 16:23:29 GMT

The 8.1.0 Interface is terribly buggy. I would advise EVERYONE to not use it until they at least patch it once.

NAT and Security rules do not highlight correctly at intervals, and it has already cost us production time as we were attempting to modify one NAT rule (after highlighting it) and having it return a different one. All browsers show this issue too.

Its a bad release, and I wish PA would have done a better job of QC'ing it instead of expecting the rest of us to "Fix the airplane before it hits the ground".

Ususally PA is good about this. But this one is a clear miss, and the release should be pulled.

Re: Quick Note on 8.1.0 Deployments

Brandon_Wertz — Mon, 23 Apr 2018 16:25:36 GMT

@ITSysEng wrote:

Its a bad release, and I wish PA would have done a better job of QC'ing it instead of expecting the rest of us to "Fix the airplane before it hits the ground".

Ususally PA is good about this. But this one is a clear miss, and the release should be pulled.

Then there was 7.0.0 which was totally deferred...I think 7.0.0 still wins this "contest" haha

Totally agree I've always thought the major releases process could use a bit of improvement. No doubt there's push to deploy new capabilities from Palo, but like has already been mentioned; if as admins of a service, in this case a company's edge/firewall environment the onus is on US as firewall admins to do the due diligence to ensure the code version is stable and appropriate.

This includes dev/qa testing if need be. If an admin wants to risk their career because a new software release is out that's on you. Not Palo IMO.

Re: Quick Note on 8.1.0 Deployments

CaviumKeith — Mon, 23 Apr 2018 16:30:09 GMT

I'm hearing a target of early May for 8.1.1 roll out, so just a couple more weeks.

Re: Quick Note on 8.1.0 Deployments

BPry — Mon, 23 Apr 2018 17:32:03 GMT

@Brandon_Wertz,

I think you nailed my reaction on all of these "PAN-OS 8.1.0 is real bad; Palo should pull the release; this killed my production network". Palo Alto did not come on premise and force you to upgrade to 8.1.0, Palo Alto didn't stop supplying updates to other software versions, they didn't release hardware that only runs 8.1.0 (unlike 8.0), and Palo Alto didn't automatically install 8.1.0 to anyone's firewall.

I agree that Palo Alto should, and could, tone down the marketing around 8.1.0 until its actually hit recommended status. There are defiantly mistakes that they've made in that regard. There shouldn't have been as much noise surrounding the 8.1.0 release, the SMB over VPN issues experianced by users of the Beta channel should have been included in the release notes, as it was already noticed by the time 8.1.0 was offically released, and it should have been clear in all communication that it wasn't recommended you actually install it on production equipment.

In the end though it comes down to the decision that you made as an admin. You choose to run 8.1.0, you made the decision to run new code, you made the decision to prioritize features over stability, you failed to QA the code with your configuration to verify it didn't effect your production network. YOU caused an outage or a degradation of services because you installed 8.1.0 on your production equipment without validating it worked for you and your company.