Friday Sep 03, 2010
Unit tests rendered useless
In this post I will argue why test coverage obtained from functional tests leads to associated metrics from which genuine conclusions can be drawn, as opposed to the much more commonly accepted view that useful test coverage related metrics can be obtained from mere unit tests. Moreover
For almost a year now, we have been putting almost all our testing efforts in writing functional tests, as opposed to the more classic and commonly accepted (j)unit tests. In this context, functional tests are purely derived from and based on user stories.
Primary motivation was and still is: the customer is primarily interested in the fact that his user stories are working satisfactory, and keep on doing so during development, considering the necessary and frequent refactoring efforts that are a direct consequence of the build the simplest thing that could possibly work principle from agile software development.
Stated differently, what does the customer care about a unit test coverage of 90% (or supposedly even higher), while at the same time his product is still full of flaws? This is almost always the case, as (j)unit tests are necessary but not sufficient (if all the parts work separately, they do not necessarily do so when tied together to form an application).
On the other hand, inspection of the line coverage based on functional tests must lead to the conclusion that code that isn't covered is either caused by the fact that the code is redundant (dead) or the test does not cover all aspects of its associated user story. Practice has shown that this leads to a typical line coverage of around 95%.
Last but not least: functional tests makes TDD much easier. Write your functional test(s) first, based on your user story, and implement the simplest thing that could possibly work thereafter, until your test(s) succeed(s).
Summarizing, we found the following compelling arguments in favor of functional tests: