topic Re: QA new release in General Topics

QA new release

kevin_thys — Fri, 12 Apr 2013 15:06:44 GMT

Hi,

Anybody having much issues with latest release on 4.1.11 and 5.0.2,3,4?

I think quality assurance should be increased. As we have a lot issues with those new releases.

=> user id taking 100% mgmt on 4 and 5 releases.

=> connection to user-id agents flapping

=> custom url blocking page cause client browser to freeze

=> app override causes sessions to be removed after 10 seconds

=> interface stays down on PA-2020 device after upgrade from 5.0.2 to 5.0.4.

=> 5.0.4 on PA-2050 webinterfaces "dies" need to restart service in cli.

I should think those issues could be detected during testing before they release a version. This are not special setups that are diffucult to test.

Had to share my frustration over here after a heavy week of issues....

Regards,

Kevin

Re: QA new release

darren_g — Wed, 17 Apr 2013 02:31:17 GMT

General consensus is PA has sacked their entire QA department and replaced them with rejects from Microsoft.

I certainly feel your pain - gone down this road a fair bit lately. My main reason for not upgrading from 4.1.11 is that I can't trust 5.0.x not to break my setup in a huge way.

This didn't happen in the days of 3.x - perhaps because PA were not being pushed by other vendors products moving into the "Next generation Firewall" space, and hence weren't rushed into releasing features which weren't ready - but I've lost some faith in PA as a result of the recent issues with what is plainly piss-poor QA.

Re: QA new release

mikand — Wed, 17 Apr 2013 05:23:46 GMT

Actually its PA who has defined what a "Next generation firewall" is according to Gartner.

But sure now when the competitors noticied they need to call their stuff "NGFW" aswell (even if they in many cases isnt a NGFW) then PA seems to have been stressed in a bad way.

But I agree with kevin.thys - Im too a bit worried about the degree of bugs and malfunctions that are being found in the PA product line (or rather PANOS line). Or rather that these bugs/malfunctions are still there in RELEASE code and in multiple cases also still present even if the devteam claims its fixed.

Im thinking of the lately QoS-bugs aswell as the 100% cpu on the mgmt-plane.

Would be interresting if some official comment could be made from a PA representative in this thread regarding how the QA process is or will be improved. Whatever was done last summer (when it is claimed that developing was halted and focus was put on fix issues/bugs in currently releases) doesnt seem to have been enough.

And currently things like this will be used as an argument against choosing PA which I find is troublesome (specially in situations where PA would replace another vendor in a design where it in many cases is easy to forget current issues with the current vendor for those who like that vendor for one reason or another, for example that Checkpoint cannot deal with NTLMv2 Identity Awareness Wizard authentication fails is put aside because "AppID in PA doesnt work anyway" according to the youtube video Checkpoint presented last christmas (even if PA was fairly fast to fix this issue)).

Re: QA new release

darren_g — Wed, 17 Apr 2013 22:20:39 GMT

mikand wrote:

But sure now when the competitors noticied they need to call their stuff "NGFW" aswell (even if they in many cases isnt a NGFW) then PA seems to have been stressed in a bad way.

That was kinda my point - back in the 3.x (and earlier) days, PA *had* no competition in the NGFW space, so they could take their time with upgrades, and make sure they worked properly.

Today, every mongrel and his dog is calling their stuff a "NGFW", and PAN obviously feel they have to rush new features to market so they can keep waving the flag and saying "See? We're the innovators here. These other clowns are just following us!".

mikand wrote:

Im thinking of the lately QoS-bugs aswell as the 100% cpu on the mgmt-plane.

Yeah, I've been bitten by a couple of nasty ones which simply should not have made it through QA - any QA engineer worthy of the name would have tested the functions concerned in a real-world environment and found them before referring them for rectification.

mikand wrote:

Don't hold your breath waiting for this one - seems these forums, while scattered with a sample of genuine PA employees, aren't "officially" monitored, or if they are, are deemed not significant enough to be worth commenting on.