gecko-dev/webtools/tinderbox2/README.TESTING_IDEAS

Automated Testing


The tinderbox framework makes it ideal for displaying a wide variety
of metrics about the project's health. I have worked in companies 
examined metrics only at specified milestones. Between the milestones
the development work would incrementally erode the metrics and slowly
slip the quality of the product.  The result was that when the next
milestone was reached and metrics were finally examined major rework
was needed to bring the code into compliance.  Tinderbox makes it very
cheap to continual monitor a wide variety of metrics and identify
incremental problems with the code.  It is much easier to keep an
application in compliance with a set of metrics than it is to "bring
an application into compliance".

While tinderbox can not reduce the coding effort needed to use a large
number of metrics it does reduce many of the management problems with
metrics.  It reduces the expense of generating metric reports and
communication problems in disseminating the information and in viewing
the time when checkins improve or reduce the metric's quantity.  It
reduces the time between when a developer checks in code and the group
is aware of the results of the check-in on global quality.  By
shortening the feedback loop and making it visible to everyone it is
possible to encourage better development practices.

There are numerous types of automated tests which tinderbox can
monitor.  A wise organization will consider a diverse set of tests to
monitor different aspects of its development.  Here is a short list of
automated tests which easily fit into the tinderbox framework and
would be of benefit to a wide variety of software projects.

One goal of version control is reproducibility.  Simply stated it
should be possible to make small source code changes to the code which
was built last year and recompile to get a nearly exact duplicate.
When faced with rebuilding code which has not been looked at in a long
while, there can be missing build information which prevents an exact
duplicate from being built.  Often it turns out that not all the
source code was saved into the version control system or that there
are several manual steps in the compilation process which could be
completed in different ways. By the time that these missing data are
identified information about their state at the time of the original
build has been irretrievably lost.  It is common to require the code
be checked out in an automated fashion and that code compiles with a
single standardized command without manual intervention.  This test
needs to be done on a standardized machine which has well known tools
installed. So it is a common that developers do not have the ability
to change the tool set on the compilation machines but they need to
see the results of the attempted compilations. The reporting of build
failures to development can be a communication difficulty. Tinderbox
makes administering these compilation machines easier.  Developers
get the information they need via the web however the compilation
machines are maintained by the QA group.  Each developer is
responsible for ensuring that their changes compile. This eliminates
the dreaded "build master" job of integrating broken code.  Many QA
departments use the automated builds as the 'official binaries' and
make these available to the developers via network filesystems.

Portability tests are the bane of large scale development.  Often not
every developer codes on all platforms.  Programmers are often unaware
when they introduce portability problems.  Tinderbox provides a
visible means of getting quick feedback to all developers about the
current portability of their code. This also prevents developers from
unknowingly checking in code which does not work.  The tinderbox
framework is a convenient place to assimilate the information from
multiple architectures.  The machines send the build logs to the
web-server and the developers access the logs via web-pages. This neatly
solves the typical problem of giving multi-platform access to a
compilation farm. All platforms can send mail and post web-pages so the
administration of the data passing to the tinderbox server is
easy. The tinderbox server organizes the large amount of log data from
all the build machines so that the results are easy to find and view
from any web-browser.  The build machines can have restricted access
policies since all the build data is available via the web there is no
need to log into the machines.

Too often in projects architectures degrade over time.  Modules become
entangled in nets of dependencies and any code structure becomes
lost. Architectural violations can often be detected at compile time
with a bit of thought.  These tests are often as simple as checking
out a subset of the whole application and seeing if it will compile
into an independent module.  For example network code is often viewed
as being very low level and database code very high level.  If a
program's networking module was to use code from its database module
this would most likely be an architectural violation but the case of a
database module using networking modules would be expected. The
architecture can be enforced by checking out only the low level code
and ensuring this compiles in isolation. The high level code can be
checked out with the low level code since we are not concerned about
its ability to work in isolation.  More sophisticated tests will
ensure that all modules implement a previously defined interface
and that interface is the only means by which data is shared.  I find
it useful to put the official interface definitions in a separate
directory where they are not mixed with the code. Some interesting
discussions of Module Dependency issues and independent compilation
can be found in "Large Scale C++ Software Design" by John Lakos.

There are numerous 'lint tests' which could be run on the source code
to ensure that developers have followed all the coding styles which
are in effect and that static analysis of the code does not reveal any
ambiguities. The mechanics of this is very similar to automated
compilation.  Code is checked out and tools run on the sources.  The
results of the "compilation" are mailed to the tinderbox server with
some indication as to whether success or failure was recored.
Developers need this coding style information as they check in or they
can quickly accumulate enough lint errors that upgrading the code to
lint compliance is an onerous task.

Programmers can often write unit tests to test the functionality of
individual modules.  The XP development movement is particularly
interested in developer created automated tests. It is common in our
industry to monitor the percentage of code covered by unit tests and
require a certain percentage of code be executed by the automated
tests.  This will ensure that as the code base grows the developers
grow the unit tests to match it.  Similarly applications may code
performance tests which measure application specific timings and
ensure that recent code changes do not make these timings worse.
These automated tests are then easy to integrate into this framework.
Tinderbox provides a convenient display for the history of the success
of tests.