Testing without it being a trial

Testing device agents in a multitude of platforms and environments can be quite a challenge. Long hours, constantly repeating the same tests every time a developer makes a change — not anymore!

But first some background. We have a complex environment.

The Appweb and GoAhead embedded web servers are used on many platforms from PCs to embedded systems. They are deployed on a wide variety of operating systems such as Linux, Windows, VxWorks, FreeBSD, MAC OS X including iPhone OS and other embedded kernels. We support many CPUs including X86, X64, MIPS, PPC and Arm. We also provide these products in source code form where users can configure the required feature set by running a configure program to enable and disable features and modules. This quickly multiplies to yield to a very large set of combinations.

The question is how to best test this and deliver high quality, robust products?

Continuous Integration

Continuous Integration to the rescue. This is the practice of immediate and automated integration and testing of all changes to the source code base. It typically involves creating a Build Farm of client systems that are controlled and coordinated by an automated build server. The Build Farm detects changes to the source code repository and the server instructs the client systems to get the new source code and build and test the new code base.

We could have used one of the existing continuous integration systems, but none seemed to work very well with embedded platforms nor with the extent of our supported configurations. Fortunately, we wanted to test and show-case the ESP web framework, so we wrote it completely using Appweb and ESP. While it is not a complete product, it turned out quite nicely for a little demo app.

Small, Fast and Flexible

The client code that performs the actual code checkout, build and test is only 308 lines.