Technology weblog

Friday Sep 03, 2010

Unit tests rendered useless

In this post I will argue why test coverage obtained from functional tests leads to associated metrics from which genuine conclusions can be drawn, as opposed to the much more commonly accepted view that useful test coverage related metrics can be obtained from mere unit tests. Moreover

  • Functional tests render most mocking frameworks redundant.
  • Line coverage of 95% should be feasible without too much effort.
  • This approach is seamlessly applied to agile development techniques.

Functional testing

For almost a year now, we have been putting almost all our testing efforts in writing functional tests, as opposed to the more classic and commonly accepted (j)unit tests. In this context, functional tests are purely derived from and based on user stories.


Primary motivation was and still is: the customer is primarily interested in the fact that his user stories are working satisfactory, and keep on doing so during development, considering the necessary and frequent refactoring efforts that are a direct consequence of the build the simplest thing that could possibly work principle from agile software development.

Stated differently, what does the customer care about a unit test coverage of 90% (or supposedly even higher), while at the same time his product is  still full of flaws? This is almost always the case, as (j)unit tests are necessary but not sufficient (if all the parts work separately, they do not necessarily do so when tied together to form an application).

On the other hand, inspection of the line coverage based on functional tests must lead to the conclusion that code that isn't covered is either caused by the fact that the code is redundant (dead) or the test does not cover all aspects of its associated user story. Practice has shown that this leads to a typical line coverage of around 95%.

Last but not least: functional tests makes TDD much easier. Write your functional test(s) first, based on your user story, and implement the simplest thing that could possibly work thereafter, until your test(s) succeed(s).


Summarizing, we found the following compelling arguments in favor of functional tests:

  • Inspection of test coverage metrics lead to conclusions that have business value, since you either missed to test an aspect from your user story, or your code does nothing and can be removed.
  • There is no risk of writing unit tests for dead code (I have seen this happening all too often).
  • As the functional tests are coded, regression testing comes as a bonus.
  • Line coverage higher than 90% can (and should!) easily be obtained with functional tests, whereas this can hardly be met with pure unit tests, i.e. without tricks such as reflection (e.g. for getters and setters).
  • Refactoring is no longer an issue, as one can refactor code as much as needed, as long as the functional tests refrain from failing.
In a subsequent post, I'll elaborate on the details related to the technical realization.


Post a Comment:
Comments are closed for this entry.

Hire us