Being able to rely on a test suite during software development helps to find bugs, prevent regressions and improve code quality. The often-quoted “80% target” is an ambitious one, but how do we test the quality of our tests while reaching it along the way? In this article, we look at one answer: mutation testing and how to “kill the mutants”.
- Author
- Dominic Luidold
- Date
- August 2, 2023
- Reading time
- 3 Minutes
Code coverage, metrics and a false sense of security
When you talk about software development projects, at some point you will be asked the question: What is the test coverage? Depending on the answer, e.g. 75%, 80% or even 90%, you may not take a closer look at the thousands of lines of tests in an already large project.
The problem with that: PHPUnit, for example, offers various different metrics to measure the code coverage of a project:
- line coverage
- branch coverage
- path coverage
- …
Each metric measures coverage in a different way, which may allow you to sleep soundly if your personal or project target is met, but is difficult to compare or to draw conclusions from. At worst, it can give you a false sense of security.
Mutate your tests, kill the mutants
A solution: verify your tests and especially what they test by mutating the source code. The Infection framework allows you to do exactly that whilst re-using your existing PHPUnit, PhpSpec, Pest or Codeception test suite.
Mutation, mutants? Once installed and configured, the framework alters the source code of your project (creating a so-called mutant) by applying different mutators and subsequently using your existing tests to verify if the test covering a specific function or feature fails or still passes.
A quick example. For the following code…
public function isInRange(int $foo): bool
{
if ($foo <= self::BAR) {
return true;
}
return false;
}
…there are several different mutants:
// Mutant one
if ($foo < self::BAR)
// Mutant two
if ($foo >= self::BAR)
// Mutant three
if ($foo > self::BAR)
The result of running Infection are three different metrics, of which the Mutation Score Indicator (MSI) is the most important one. The MSI tells you how many of the source code alterations (mutants) were detected by a failing test, “killing” the mutant.
Having a code coverage of 76% whilst having a MSI of 65%, for example, shows you that the tests might need some further fine-tuning to improve the overall quality.
Caveats and what to expect
You may have asked yourself by now, isn’t dynamically mutating the source code of a large project resource and time consuming? It certainly is, regardless of the environment in which you run Infection, e.g. locally or in a (continuous integration) pipeline. While you could theoretically run the mutations on your entire code base, it makes most sense to run them only on a specific subset, e.g. for specific changes in your merge/pull request, using the configuration options provided by Infection. Also, you might want to consider only running the mutation tests on your unit tests, as tests are executed randomly, which might cause issues with functional tests.
In addition to that, there is a learning curve to interpreting the results of the console output. To improve the experience, make use of the various log formats, such as the interactive HTML report.
Conclusion
Having a test suite with a high code coverage is great. Nevertheless, you shouldn’t solely rely on the coverage percentage and also use other tools, such as mutation testing, to verify the actual quality of your tests.
The concept also isn’t unique to PHP. Stryker.NET offers a similar framework for testing .NET Core and .NET Framework projects, while there are options for Java and other languages and frameworks as well.
