Unit Test Platform

[instance]

CirTap is currently working on enhancing the platform we're using for unit tests. It's using SimpleTest (http://www.lastcraft.com/simple_test.php) as the core. CirTap is adding some capabilities to group tests, and to run subsets of tests so that the tool is as useful as possible to developers.

The next big job is mock classes. Ideally we can have a library of fairly standardized mock classes, possibly more than one per real class, depending on what needs to be exercised. We're also looking at a special version of jimport() that can be told which classes to load (real or mock, and if mock, what version).

Once the general architectural direction is defined, the real work of building mocks and tests will get under way. As it stands, we have close to a "clean slate": very few classes have test cases. We're basically going to start walking through the framework classes and writing tests until we've got a reasonable suite (I'm going to tackle the filesystem group of classes first). Once the framework is done, everything in the core is fair game.

If all goes as it should, developers will soon realize the exceptional value of a good test suite and start contributing their own tests. As bugs are found, Q&T should create regression tests for them if possible. Aside from no bugs at all, there's nothing better than a bug report complete with an exemplar test case!

We're also discussing the idea of some sort of test result repository (in very general terms, this is just a dream at the moment). In an ideal world, we have users out there with a variety of unique configurations who we can tap as "designated configuration volunteers" for running the test suite, so that if there's a problem that only shows up in a Linux 2.6 kernel under Apache 2.2 with PHP 4 running as a CGI and using MySQL 4.0, we'll know about it sooner rather than later.  If we can have the test suite report results back to a repository via XML-RPC, we should be able to post the results publicly (excluding security issues). This way people with unusual configurations can at least identify the problems they're likely to have right up front. Hopefully this will reduce the number of support issues as well.

[CirTap]

The bazaar is open!

Thank you Alan for this nice announcement :-) I hope the changes will become useful.

So, what's new?
We're now using Simpletest 1.0.1 .. despite labeled as beta it's working well, and we may not necessarily run into any of the bugs reported at Sourceforge.

The major change affected the testfiles, their names, and the general "logic" in how single tests and groups are run.
Many framework files do contain more than one class, but the "old" skeleton builder didn't honor this which would result in some classes never beeing created w/o manual intervention. The new builder (based on PHPUnit) will find *evey* class in a source file and creates as many testcase skeletons, using the original classname as the file's basename with a "Test" prefix, ie.
  factory.php => class JFactory => JFactoryTest.php
  router.php => class JRoute, class JRouter => JRouteTest.php, JRouterTest.php

The creation of new skeletal testcases has been restricted to PHP5 (CLI) due to the use of the Reflection interface. There's *no* intention to make this work under PHP4, don't even ask. I blatantly presume that every one who's able to write a new testcase for this framework will also be able to run PHP5 on the command line...

Each subfolder (subpackage) will now contain an "AllTests.php" file to bundle a distinct set of TestCases and Suites. These files do *not* simply include anything they can get a hand on, but use a "hand crafted" list of tests ordered by significance. There are dependecies among framework classes, and they need to be addressed in the testsuites as well, simply put: if JFactory should fail many classes will be "broken", thus it woudnt make much sense to continue with any test that relies on the factory class.
Since we're at a very early stage, the AllTests are not very impressive and any dependecies need to be evaluated and implemented as they come.

Although prepared, but not yet functional, is the ability to configure each and every testcase whether it should run on your machine or not, e.g. if you can't test for JFTP or JLDAP, you may simply disable them in your (new) local TestConfiguration.php. This file currently hold a vast amount of constants built semi-automatically from the available classes in the /libraries/ folder. Whether they'll all be required or replaced by some "registry" is subject to change. This registry will also be used to allow both, renderers and testcases, to prevent subsequent/dependent tests from running ... at run-time -- ommited tests will of course be flagged as a failure.
This should allow smooth runs w/o faking the allover score -- i.e. you won't get a "green" by simply disabling all tests

That's about it for now. Please check the included README's and TODO's, and feel free to ask, suggest, comment, ...

Have fun,
CirTap

[CirTap]

Hi,

it's been a while since this threat was started, everbody's been busy like hell. Some with coding, some with making money ...

I found a few spare moments in the last weeks to add some shape to the "platform". Most of it was rewritten from scratch. It's far from complete, but the html-renderer are working (for me).
Consider the code available in SVN alpha quality

Please check the README.txt and CHANGELOG.txt

Tests implemented:
- JFactoryTest.php
- JFrameworkConfigTest.php
- JFTPTest.php (thanks to friesengeist)
- JObjectTest.php
- JVersionTest.php

As a skeleton example (still):
- JArrayHelperTest.php

Some Tests fail  ... sorry, I mean, there are issues that should be ironed out, like dependent classes not being loaded if separated from the CMS / Application -- but I might be missing something.

@todo: a lot (see the README) incl. a full TestSuite for the UnitTest Platform

Have fun,
CirTap

I recommend the "custom" renderer.