Saturday, 15 November 2008

Flex testing frameworks

Over a month ago, I was asked to do a Flex programming exercise to demonstrate my ability to program. They specifically asked me to use the FexUnit testing framework, and the Carngorm model-view-controller (MVC) framework. In my usual way of investigation, I explored the range of possibilities for testing frameworks for Flex. I did look up the MVC and MVP (model-view-presenter) options but only experimented with Cairngorm.

There are a lot of what I would call “me too” testing frameworks available. Some are little more than announcements of intent to produce one. The one's, that I will review here, were selected because they indicated some unique features rather than saying that they simply implemented the JUnit style framework. The frameworks looked at included FlexUnit, fluint, FUnit, and FluxUnit.

What are my credentials for being able to do such an evaluation? Some thirty plus years ago, I started my career as a programmer in the computer industry. One of my early projects was to write a terminal simulator so the programmers could test their programs without having to have access to the networked terminals. On the same project, I was also involved in building what might now be called an integration tool. The team that I worked in was responsible for support routines that needed to be tested and released for the application programmers.

Over the last nine years as a lecturer in a university, I have been working with and teaching test-driven approaches to programming using both JUnit and NUnit. When I start a new project, one of the first things that I look for are testing frameworks for the languages that I will be using. Also for the DotNET environment, I wrote my own database test unit (DBUnit) and a Forms testing extension for NUnit as a result, I feel that I can talk with some confidence about testing frameworks and what they have to offer.

FlexUnit

I am going to use Adobe FlexUnit as the base testing framework since it is distributed by Adobe through their open source sight and appears to be fairly widely used and supported in the Flex community. It, along with dpUnit, the forerunner of FlUnit, was the first testing framework that I experimented with in the Flex environment.

FlexUnit is based on the original JUnit / SUnit (Smalltalk testing framework) design. It doesn't use reflection but requires the programmer to build suites of tests that are then passed over to the runner. This management code is tedious to write and it is easy to miss adding a test to the suite. A test can be another suite of tests or a refrence to a specific test. You have to be disciplined to ensure that you set up this management code correctly. The usual practice is to put a class method called suite at the front of all the classes that implement tests (i.e. extenf the TestCase class). This method simply adds the tests to a suite that is returned to the caller (i.e. 'theTesSuite.addTest(new testClass(“testName”);'). You then create one or more suites in a hierarchy that enable all the suites to be passed in one batch to the runner.

 var testSuite: TestSuite = new TestSuite();
testSuite.addTest(testCase.suite());
testSuite.addTest(testcase2.suite());
testRunner.test = testSuite;
testRunner.startTest();

For those used to the early versions of JUnit, this conforms to the practice that was then required to set up the test suites.

The avalable asserts also follow the original pattern of assertEquals, assertNotEqual, etc. These are all provided in the TestCase class which you inherit into your own test classes.

There is also available a FlexUnit Ant tasks and the Flex Mojos that can be used to integrate FlexUnit tests into an Ant build or Maven build process. I haven't tested the Ant tasks but have used the Flex Mojos to build and run tests.

Fluint

This was originally called dpUnit and is developed by digital primates IT Consulting Group. Fluint claims a number of advantages over FlexUnit including support for asynchronous tests and the testing of user interface components. At this point in my experiments, I haven't used these advanced features but they are on my list of things to experiment with.

In many respects, fluint follows the example of FlexUnit. For basic testing, one of its advantages is that you don't have to create a suite that contains the individual tests. It uses reflection to identify the tests looking for method names that begin with test or the [Test] attribute on a method. My experience so far indicates that the use of attributes hasn't extended to the test class (TestFixture) or the setUp and tearDown methods. Experiments are ongoing.

Fluint also uses a slightly different approach to building the base suite.

 var suiteArray:Array = new Array();
suiteArray.push(new testSuite());
testSuite.addTestCase(new testCase());
testRunner.startTests( suiteArray);

In many respects, this is cleaner than the process used for FlexUnit. In terms of the base assertions used by fluint, they are identical to FlexUnit. The asynchronous testing seems to be a new set of methods that don't fit the traditional assert structure. I will report on those once I have conducted some experiments.

Fluint also provides Ant tasks and with Flex Mojos abilty to define your own test runner, I suspect that it would be possible to integrate fluint into a Maven build since the Flex Mojos use the FlexUnit Ant tasks to run the tests and Maven allows Ant tasks to be integrated into the build.

At the basic level, fluint, simply provides a different way of setting up the test suites. If it has a real advantage, it has to be in the support that it provides for asynchronous and user interface testing. You may have noticed that the name isn't FlUnit but fluint. The authors argue that it is aimed at integration testing.

FUnit

FUnit follows the more recent versions of NUnit in the way that it identifies tests and in the structure of its asserts. It uses a tag-based attribute model to identify test classes and the tests that they contain. This means that there is less set up coding that needs to be done to make set up the test suites. The test classes still need to be added to a suite but not the individual tests. I am not going to illustrate its coding pattern here as I don't think it differs enough for fluint.

It has one major drawback and this is that in its current version the GUI interface doesn't run the tests. It has to be run through a console runner and as at this writing, the only way that I have been able to view the console is running the test program in debug mode within the Flex Builder. Not exactly helpful. However, the next version of the Flex Mojos will support the running of FUnit tests in a Maven build so it may be useful in that context. What is available in the library presents a nice user interface but doesn't report on the tests. Once they have this implemented, this could be a good package.

The assert structure in FUnit follows the more recent structure introduced in NUnit. That is it uses an Assert class with static methods (i.e. Assert.areEqual()). This has the potential of allowing the user to easily extend the conditions that are available or even the more recent enhancements that have been introduced in NUnit.

FluxUnit

This claims to be a behaviour-driven (BDD) framework rather than a test-driven (TDD) framework. As a consequence, the way that it implements its tests is very different to the other suites covered in this blog.

FluxUnit uses describe, before, it, and after methods and dynamic classes. I am relatively new to Flex so some of the coding patterns used in FluxUnit are novel to me and I will not attempt to explain what they are doing. The code to run the tests is as follows. Note that the creation of the object containing the tests isn't explicitly assigned to anything.

     Flux.setRoot(body);
new CRFluxPeriodOverlap();
Flux.Run();

In terms of defining a test, this looks quite messy to start with but become easier as you gain experience.

     public dynamic class firstSpecs
{
public function firstSpecs()
{
var self: firstSpecs = this;
new Flux(this).Unit(function():void {
with (self) {
describe('initial state', function():void {
before(function():void {
// setup code;
});
it("expected result", function():void {
expect(varaible).to(equal(),value);
});
after((function():void {
// clean up code
});
});
}
});
}
}

This appears to work by defining dynamic functions. It is possible to nest describes and to have before and afters that are global to all the describes. The user interface reporting is also interesting since it uses the text from the describe and it functions to structure the output.

The authors have an example of using ot for asynchronous testing but not for user interface testing. There is also no reference to Ant tasks or ability to work with Maven Mojos.

Although I like the terminology used in behaviour-driven development, I still find the tools uncomfortable to use. At least this one is workable if you apply a simple template. It is however, to misplace the bracketing and take some time to get it corrected.

Conclusion

At the basic functionality level, there is little difference between any of these frameworks. FlexUnit appears to be a proven framework and is able to be used in Ant and Maven builds. FUnit appears to have Maven support and possibly also could be used in Ant builds but its lack of a GUI is frustrating. Fluint has the additional functionality for asynchronous tests and user interface component tests. It also has Ant support and may be easily integrated into a Maven build. FluxUnit provides more readable output.

My personal order of preference at this point is fluint and then FlexUnit. I would use FUnit ahead of FlexUnit if the graphical user interface worked. I intend to keep experimenting with FluxUnit simply because I like the BDD style but until I see support for Ant and Maven builds, I wouldn't use it on a project. I would also like to see the coding structure improved so that it was easier to set up and use.

2 comments:

Ryan said...

Errol,

Thank you for providing such a detailed contrast analysis of the major Flex unit-testing frameworks.

As creator of the FUnit testing framework I was honored to be included in your study. Your feedback is invaluable and greatly appreciated.

Admittedly, advancement of this community project has been a tedious undertaking and limited to free time spent in the late evening.

Although the GUI is an upcoming feature in the FUnit Roadmap, it's clear that I underestimated it's need even at the early development stages.

Please contact me here if you have further inquiries or suggestions. I would also like to inform you of future updates and changes.

Sincerly,

Ryan Christiansen

Michael Labriola said...

Thanks for the write up. I am responsible for Fluint so, if and when you have the time to check out the async stuff, let me know.

Would be glad to steer you through that evaluation.

Cheers,
Mike