Extend Beta testing time Topic is solved

[LukeDouglas]

I really feel sorry for the Joomla development team concerning the critical bug introduced in 3.9.9 and corrected in 3.9.10. I've used Joomla since the split from Mambo back in 2005 and these issues are exceedingly rare. The Joomla development team (as 'all' CMS development teams) are NOT perfect. We are all human.

I can only suggest to maybe extend the Beta testing period.

Keep up the great work!

[deleted user]

The problem is not in the length of time in having a "dedicated" RC phase for a patch release. The problem is nobody is testing in general, or nobody is testing against real life configurations.

The problematic patch landed in core on June 16. It has therefore been in every nightly build since then. Meaning it was included in the release candidate. Not one person reported the problem, which most likely means not one person tested a nightly build or the release candidate with a multilingual website. The fact it went unnoticed for 3 weeks means adding more time to a "dedicated" testing window wouldn't fix anything.

You want to fix the release problems? Campaign the project to get serious about proper QA teams. Campaign the project to get serious about having proper automated test coverage for the majority of the application stack (I'll be kind and say maybe 20% of the Joomla owned code in the ZIP package has test coverage, emphasis on Joomla owned because I'm purposefully not trying to mix in test coverage from third party vendors (PHPMailer, Paragonie, Symfony, or TYPO3) in that metric). Campaign those who actually do take the time to test to test against real life configurations, and not just "download a package, install it, and call it good". The majority of the FUBAR releases are FUBAR because people are pencil whipping the testing and QA of pull requests and releases in general.

[LukeDouglas]

mbabker,

Has this issue been degrading over the years? I believe there was more people involved in past years. As far as downloading, installing and calling it good, I'm a bit more involved as clients have stayed with me for my dependability. I test everything which is how I found issues with older versions of CSS coding no longer being recognized and throwing errors. In addition, I just found an issue with the Owl Carousel particle but I'm not sure it had anything to do with the Joomla update.

I will try to test future RC releases on some of my 'personal' websites. If they get trashed, it doesn't affect the bank.

[sozzled]

Knowing that whatever opinion I may have and that I'm going to be criticised for it, I enter this discussion with hesitation. I want to begin by acknowledging my deep respect for both the OP (for raising the matter) and to @mbabker for responding in a timely manner.

The last couple of releases of Joomla have had unfortunate consequences for owners of multi-language websites. I don't fall into that category and, therefore, I'm unqualified to comment on the adverse effects that have affected a sizeable number of members of the community.

As a general rule, I approach the process of software update/upgrade fairly cautiously. I have a few test sites where I trial the new software and I run a few simple tests. My testing is not exhaustive. Sometimes people discover new defects; sometimes the new update contains regression faults; sometimes the new software just plain falls in a heap. As the old saying goes: "If builders built houses the way that programmers build software, it would only take one woodpecker to come along destroy civilisation as we know it."

So, generally, I wait and observe the running commentary from ordinary folk who, when they discover a new software release, install it and share their experiences. I take it as a given that software changes generally only require a couple of people to agree that the code changes have caused no problems but, considering that people are human (and software developers are human, too), no-one is perfect.

There have been 72 releases of J! 3.x (starting with J! 3.0.0 nearly seven years ago). You could probably count, on the fingers of one hand, the number of times when the production team had to quickly intercept and rectify a problematic release.

Examples include J! 3.1.0, 3.1.2, 3.1.3, 3.2.6, 3.3.2, 3.8.9, 3.9.7 and 3.9.9 ... OK, so I need more than one hand

But that's not a bad result over the past seven years, is it? About one in 10. And these weren't unmitigated disasters; they were just the result of human error.

Now that I'm well into the seventh decade of my life, I don't have the same level of enthusiasm as I once had in my twenties for hurling myself into alpha testing. For that reason, I've steered clear of the armchair discussions about the design of J! 4. I'll leave that to the pointy-headed sideline commentators, who run onto the field of play whenever they feel like it, to argue and debate (and delay) the playbook for a project that will soon enter its third year.

It's open source, for goodness sake, and the product is only as good as the collaborative teamwork put into it by the community. Instead of picking things apart (or agonising over whether something should be coloured a particular hue of blue or have elements occupying 3 pixels more or less space than something else), perhaps the armchair experts ought take a breather, clear the air rather than pollute it?

I freely confess that I'm as guilty as the next person who doesn't invest my time, heroically and thanklessly, on QA-ing each and every release of J! I'm also mindful that a sizeable cohort of the community is not interested in what would benefit the whole; every day I see people complain about some new "bug" or harass the community with "please fix my problem now" . That's not why I joined this community and I know that's also not the reason that most people use this forum.

Enough written by me for the time being. Thanks if you've read this far. Cheers.

[LukeDouglas]

sozzled,

You've helped me more than once over the past 6-7 years and, like you, I 'too' am in my 7th decade of life.

Most issues with updates typically are not like what happened with the 3.9.9 update for multi-language websites. Back in the 2000's I did do QA testing on several sites after RC releases (hell, I even contributed two lines of code that eventually used to replace two lines of code in Joomla - not bragging as dozens of people have done that) but, like you, I'm just too old to get that enthused. But I am going to begin testing RC releases on one of my personal sites. I might even make it a multi-language site to catch any issues like this one. But that's about all I'm willing to invest. With a wife that has heart problems, 4 grandkids living distantly so long travel times, my website firm, and my continuing stupidity to try and improve my golf game, I don't have a lot of free time available. LOL

Hopefully some of the younger group will pick up QA testing of releases.

[deleted user]

There have been 72 releases of J! 3.x (starting with J! 3.0.0 nearly seven years ago). You could probably count, on the fingers of one hand, the number of times when the production team had to quickly intercept and rectify a problematic release.

Examples include J! 3.1.0, 3.1.2, 3.1.3, 3.2.6, 3.3.2, 3.8.9, 3.9.7 and 3.9.9 ... OK, so I need more than one hand

And because I've been involved in damn near every one of those quick turnaround releases, I can tell you first hand that there have only been three releases that would've actually broken your websites; 3.8.9 (use on a Windows server was catastrophically broken), 3.9.7 (SQL statement altering a column with JSON data could result in broken JSON) and 3.9.9 (data loss in template styles configuration for multilingual websites). I don't remember why it was decided to ship 3.1.1 so close to 3.1.0 (it's not like any of the bugs on the change list were really that critical IMO), but for every other quick turnaround it wasn't for something that would've critically impacted your site's day-to-day operation (3.3.3 was to change the container used for email cloaking back to a span instead of a div which caused some unwanted display changes and 2.5.27/3.2.7/3.3.6 was to patch a problematic line from a security fix in the update component (and that one could've waited for next scheduled release in all honesty, we only decided to hotfix that to mitigate having too many people update onto a version where the core updater wouldn't work)).

The root problem (and I too am guilty of this) has always been that human testing (when it happens) is generally done on a clean site and you aren't going to run into the vast majority of the bugs that get fixed in a release on a clean site (i.e. the email cloaking fix from 3.3.3 would only be noticed if you created an article that had an email address in it, probably wasn't a standard thing in 2014 for a lot of people and probably less so now). This is coupled with the fact that core's automated testing has minimal coverage of the CMS as a whole (everything that's extracted out as part of the Joomla Framework has a fairly good unit test suite that tests those packages in isolation, and last I remember the libraries/src folder in Joomla had something like 55-60% unit test coverage, but that's really the extent of the test coverage; notice I said nothing about components, modules, and plugins?).

There are actionable tasks that can be done TODAY to fix the quality assurance problems of Joomla releases. Adding more time to the release workflow for more human testing IMO is not one of those actionable tasks. Sadly, there has never been a great emphasis on improving QA processes or workflows in Joomla (automated tests aren't seen as compulsory as an example, I dare say almost no change should land in the libraries/src directory (since that's the only part of core with any form of test coverage) without an accompanying test (either validating a new behavior is doing what you intend it to or a regression test that makes sure a bug that gets fixed doesn't come back)).

[LukeDouglas]

mbabker,

I don't think sozzled, myself or anyone 'really' puts a lot of blame on the Joomla development team as they can only do so much testing. With the hundreds of add-on's and templates that are available as well as custom changes made for specific purposes on individual websites, it is an 'impossible' task to try and ensure that every update works perfectly on every website. That would be way too much for 'anyone' to expect.

I do agree with you that QA by individuals (like sozzled and me) are critical to finding issues 'before' a final release is done. We just need to get some younger people involved. If any younger people are reading, get one website or create a duplicate to do RC testing. If there were 100-200 people worldwide doing RC testing, the issue with this recent release probably wouldn't have ever occured.

Again, I cannot tell you how much Joomla has meant to me in my business since I standardized on it in 2005. There are not enough words.....

[sozzled]

Thanks, guys; both of you.

I didn't take the time to analyse in depth the reasons behind the few examples of release-one-day-then-fix-it-up-a-day-or-two-later cases. My recollection has been that the instances of problems (perhaps contributed because of a paucity of "QA") have been few. In any case, I doubt that "extending" the time for testing (and sign-off) will achieve much of an improvement.

In the last case we've discussed—the contamination of a nightly build early in the piece and not detected until after the event—well ... these things happen.

Well worth remembering (as I've often said) that software is "disposable technology". Our lives don't hang by a thread when software fails or when software updates go pear-shaped. I know from experience that failures with operating system updates are just some of the hazards we have to deal with. It's a pain when you apply a patch to, say, Windows version "x" and the hard disk gets scrambled and you have to rebuild the computer. Leaving aside deliberate and intentional changes (e.g. malware) that are intended to wreck one's IT assets, it's really a matter of risk management. Although the intentions of the J! production team are to minimise that risk, the responsibility for taking the risk is ours. That's why we have backup/recovery procedures.

The discussion we've had (and having) here has been thoughtful, respectfully fair and I feel contributes in a worthwhile way. It's terribly sad that there are a few other forum users who are not as tolerant, open-minded, resourceful or self-reliant; people who get out the flame-thrower instead of the fire extinguisher; people who hurl criticism upon "Joomla" for everything from losing sleep at night to burning the toast at breakfast time; people who want us to search the internet for them or delve into matters that are well-documented. They know who they are.

Cheers.

[LukeDouglas]

sozzled,

I started this thread with good intentions but, alas, the train went off the tracks. My opinion of the Joomla Development Team is still very high. Keep up the great work.

Luke