As mentioned in a previous post, at Genius.com one way we judge the effectiveness of our unit testing is by monitoring the percentage of code covered by our tests. While this is not an unusual way to measure the comprehensiveness of unit testing, the goal of unit testing should not be achieving 100% code coverage. The goal, instead, should be achieving 100% functionality coverage. While the difference between the measures may appear subtle, focusing on the correct goal can profoundly affect the way developers approach writing tests and the quality of the resulting tests.

Code Coverage Analysis

Loosely defined, code coverage is the measurement of classes, methods, and lines of production code covered by tests. In the PHP world, PHPUnit has the built-in ability (when used in conjunction with Xdebug) to generate extremely handy reports detailing code coverage. It is possible to view how many times an individual line of code was hit during test execution as well as aggregate data on the number of classes, methods, and lines tested at a file or directory level.

Functionality Coverage Analysis

Functionality coverage (not to be confused with functional testing), on the other-hand, measures the behavior covered by tests. Unlike code coverage, measuring functionality coverage is an extraordinarily difficult endeavor. Functionality is often a subjective term and is difficult to codify in a way that can be parsed and measured by automated tools and reported in a format consumable by people.

False Security of Code Coverage

For illustrative purposes, let us imagine a user class designed to perform CRUD operations:

class user {
    protected $fields;
 
    protected function __construct($id=null) {
        if ($id != null){
             $this->fields = db::getRow("SELECT * FROM user WHERE id=$id");
        }
        $this->fields = array('id' => null, 'login' => null, 'password' => null);
    }
 
    public static function getByID($id) {
        return new user($id);
    }
 
    public static function getNew() {
        return new user();
    }
 
    public function update() {
        db::execute("UPDATE user SET login='{$this->fields['login']}', password='{$this->fields['password']}' WHERE id={$this->fields[$id]}");
    }
 
    function delete() {
        db::execute("DELETE FROM user WHERE id={$this->fields[$id]}");
    }
 
    public function getColumn($columnName) {
        return $this->fields[$columnName];
    }
 
    public function setColumn($columnName, $value) {
        if ($columnName == 'password') {
            $newValue = hash($value);
        }
        $this->fields[$columnName] = $value;
    }
}

When writing tests for this class with a purely code coverage driven mindset, one might write a single test for the setColumn() method like:

class userTest {
    function testSetColumn() {
        $newPassword = 'I love IE6'
        $obj = user::createNew();
        $obj->setColumn('password', $newPassword);
    }
}

This would result in 100% code coverage for not only the setColumn() method but also the getColumn() method. However, the biggest problem is that the test does not uncover a rather glaring bug because it doesn’t actually assert anything!

Real Security With Functionality Coverage

If the developer is functionality coverage driven when writing tests, she will approach testing differently. First, a functionality driven developer won’t omit an assertion. After all, how can you assert functionality is working if you aren’t testing the result of the operation? identify the execution of the if statement in the setColumn() method as necessarily doubling the amount of functionality encapsulated by the method. The reason it doubles the amount of functionality is that it means that the code following the if statement can be executed with two different pre-conditions.

1. The if test evaluates to TRUE and the code inside the if statement executes → works properly
2. The if test evaluates to FALSE and the code inside the if statement is not executed → php error

If another if statement gets added below the password if statement, the amount of functionality would double again.

    public function setColumn($columnName, $value) {
        if ($columnName == 'password') {
            $newValue = hash($value);
        }
        if (!is_escaped($value)) {
            $newValue = escape($newValue);
        }
        $this->fields[$columnName] = $newValue;
    }

Now there are four paths through setColumn()

1. $columnName is password + $value is NOT escaped
2. $columnName is password + $value is escaped
3. $columnName is NOT password + $value is NOT escaped
4. $columnName is NOT password + $value is escaped

In the former case, a functionality coverage driven developer will write two tests to cover the setColumns() method. After writing the second test (for a non-password field), she will immediately notice the bug upon running PHPUnit. The same logic can be applied to the latter case to derive at least four tests.

Relationship Between Code Coverage and Functionality Coverage

The causal relationship between comprehensive functional coverage and high code coverage percentage is a one-way relationship. Comprehensively testing the functionality of your code guarantees high code coverage but high code coverage does not guarantee all functionality is tested. 100% coverage should be a by-product of 100% functionality coverage and thus, while you might measure code coverage, it is very important to look beyond the percentages to analyze the quality of the tests.