After upgrading to 8.1.1 VPN always on pre-logon->user-logon failings

[hshawn]

Wondering if anyone else has seen this?

We upgraded to 8.1.1 from 8.0.8 this morning and immediately had users reporting their VPN client prompting for credientials over and over again forever. The firewall says invalid username/password hwoever we can cerifiy this is not the case using the CLI test commands...

Rolling back to 8.0.8 resolves the issue but of course now we are stuck and cannot upgrade unless there is a way to resolve this. I have a case open but wanted to reach out to the community and see if anyone else with a similar config has seen this. 

Things to note:

We have rolled back to 8.0.8 and the VPN connections are working properly as expected again. This appears to be something related to 8.1.x and the pre-logon switching over to user-auth VPN configurations. Our vendors have on-demand VPN connections and did not see this issue during this window. 

Outline: 
This morning we upgraded PANOS from 8.0.8 to 8.1.1

VPN connections dropped

Users reported they could get on the VPN but the GP client would continuously prompt for credentials

VPN users showed as pre-logon instead of their authenticated usernames on the gateway

We tested LDAP AD authentication at on the CLI with success

We looked at the portal cert chain and it was still good

After upgrading to 8.1.1 we see a lot of warnings about SSLVPN Module (others reporting this on reddit)

Our VPN config has not changed in over a year and has been stable until upgrading to 8.1.1

To add insult to injury the tech support file seems to fail and only generate a 25Mb file (usually this is around >=60Mb)

GP clients are 4.1.1

Any ideas? 

[BPry]

@hshawn,

I'm not going to read much into this because you are attempting to run a non-recommeneded release in a production enviroment. 8.1 is not ready for production and should not currently be running on any production systems if you have something that is business critical behind it. 

I'm not trying to make it out as 8.1 being bad, it's exactly like 8.0 and 7.1 at this point in its release cycle, it simply isn't ready for a production network that uses all of the features of the firewall just yet. Let 8.1 sit for a bit and let them iron out some of the bugs that are present before installing it on your production equipment. 

FYI;

A quick glance at your snapshot makes it look like the configuration did not have an easy time converting to 8.1 from 8.0.*. This happens occasionally for any major update to a small percentage of those that perform the update. You have a bunch of null access-routes which generally means that during the upgrade process the XML didn't exactly update as it should, if you feel like it you could compare the running-config from the 8.1.1 upgrade and the 8.0.8 config and notice a number of statements that should be present that were missing post update or otherwise malformed. 

[Remo]

@BPry

I abolutely agree with your post. You know this, I know this and hopefully most of admins who have to administer with firewalls (as this recommendation to avoid using new major releases applies to almost all firewall vendors)...

... but for all that are not aware PAN could make it more obvious, as the marketing and all other available information about 8.1 never clearly mentions "Do not install this in production environments" and almost as important is the fact that using new hardware, which ships only with the new release, could be frustrating (because of bugs, bugs and even more bugs)

[BPry]

@Remo,

100% agree that PAN shares a heavy stake in allowing users to install a release that isn't ready without any warning. This could all be fixed if they simply implemented a Recommended Release status that everyone actually had access too, similiar to Cisco and their stared releases.

We've been asking for this collectively for years at this point and I'm not sure what PAN's reason is to keeping it locked in the internal knowledge base. I was even ore shocked when they officially launched the refreshed support site and this still wasn't available as that would have been the perfect time to actually implement it. 

[hshawn]

@BPry We are being pressured by the C level individuals to implement the new split tunneling features and after running 8.1.1 in a lab environment without issues for a while we decided to do our post implementation testing after hours and rollback if necessary (which of course, we did) The config not converting properly seems plausible and I might dig into that later and take a look.

[BPry]

@hshawn,

Makes sense. Ya I would look at the config and I'm willing to bet that you'll see the raw XML is malformated and didn't convert correctly. Even if C-levels are preasuring for an upgrade I would push back that it isn't really ready and that you may encounter issues as you move forward due to the early stage of the code level; kind of a CYA thing to ensure that they have all the proper information. 

If they make you attempt to do the upgrade again I'd also simply utilize 8.1.2 instead of 8.1.1 seeing as it addresses a lot of the bugs that were present in 8.1.1; I've been running 8.1.2 in my Lab enviroment and with my PA-220 home users without any issues at this point.