Code Coverage II

Interpreting code coverage methods and some advice about using them

In the previous article in this series, we looked at how to set up Codeception and PHP to produce PhpUnit code coverage reports. In this one, we'll see how to navigate through the report, and we'll update some of our tests and classes to improve the code coverage.

MODX logo

This article assumes that you've installed and configured Codeception and PhpUnit and created the classes and unit tests described in the previous articles. The test files and classes are available at GitHub here.

Code Coverage

In the previous article, we ran the coverage tests from the _build directory with this command:

codecept run unit --coverage --coverage-html

After running that command, look in the tests/_output/coverage directory for the index.php file and open it in your browser.

You should see a graph showing a summary of the code coverage results.

As we said earlier, code coverage is a measure of the extent to which you are testing all the parts of your code. There are a number of measures of code coverage. The ones we'll be looking at are line coverage, function and method coverage, and class and trait coverage. You should see those headings above the bar graph (though not necessarily in that order):

  • Line Coverage — The percentage of executable lines in your classes that are executed in your tests.
  • Function and Method Coverage — The percentage of functions and methods that are executed in your tests. A function or method is only considered covered if its line coverage is 100%.
  • Class and Trait Coverage — The percentage of your classes that are fully tested. To be fully tested, The Line Coverage and Function and Method Coverage must both be 100%.

The legend below the graph shows what the color coding means. The percentage ranges can be changed in the _build/codeception.yml configuration file.

Green is good, pink is less good, and red is even less good.

The _build directory contains our test files. You can see that the Line Coverage and Functions and Methods Coverage are both fairly good, but the Classes and Traits coverage is not so good. Let's see why.

Drill down by clicking on the _build link, then on tests and unit. You should see a list of our test files. They're all good except the Validator test. Click on that to see why. All the methods in the Validator file are listed. They're all good except the three data provider methods.

You can see the code if you scroll down. Lines in green are executed. Lines in pink are not. The first line of each data provider method is pink. Only the first line is marked because as soon as the test finds a line in a method that's not executed, it fails the Functions and Methods test and no further lines are checked. The data provider methods fail on all three methods.

The reason for this is that although the testing process executes the data provider methods, our tests never call them directly. The solution is simple. We need to tell PhpUnit to ignore those methods. We do that with a Codeception directive placed just above each method (note the two asterisks at the beginning of the comment and the @ sign):

/** @codeCoverageIgnore */

Edit the ValidatorTest.php file (in the _build/tests/unit/ directory). Add that line just above the three data provider methods. Be sure to use the two asterisks at the beginning of the comment.

If we re-run the coverage tests, the Validator test should be at 100% on all three measures like the other tests. Since all our tests are at 100%, you might want to comment them out in the _build/codeception.yml file just to simplify future reports. You comment out lines in a .yml file with a # sign, so that would look like this:

    # - tests/unit/*
    - ../core/model/*

We could also have removed them and/or added them to the exclude section, but you might want to test them again (by removing the #) if you add some more tests or modify existing ones.

The Class Tests

To see the code coverage reports for the classes, we need to get back to the initial report. Either relaunch the index.php file or click on the left end of the breadcrumbs at the top of the page.

Click on core, then model (or whatever takes you to the list of classes).

You should be looking at a list of the User classes we tested. The worst one is the User1 class with a 50% rating for Functions and Methods, and a 0% rating for Classes and Traits.

Notice that all the classes have a 0% rating for Classes and Traits. This is not as bad as it looks. A class gets that rating unless it has 100% on the other two measures.

Let's drill down and see what's wrong with User1 class. Click on the user1.class.php link at the left. Below the averages at the top, you see the four methods of our class. The first two, __construct and set, look great. The other two, get and save, not so much.

The save() method is the worst, with 0% across the board. You can probably guess why that is. We never tested it at all in our tests. There are two easy ways to fix this. The first is to add the directive we saw earlier just above the save() method:

/** @codeCoverageIgnore */

The directive above will take the save() method out of the code coverage testing. It won't be used in calculating the code coverage ratings at all, and its statistics will show as n/a — not available.

The other, probably better, way to fix this is to simply add a test for it in the T1_UserConstructorTest.php file. Add this code to the end of the class (just above the final closing curly bracket):

public function testSave() {
   $user = new User1();

   $result = $user->save();

It isn't much of a test, but it satisfies the code coverage gods. This is a good thing to keep in mind. Code coverage isn't everything. You could have 100% code coverage and still have bad code and meaningless tests. Code coverage is a good indicator of the completeness of your tests, but you need to think about making your tests meaningful.

Another thing to consider is that trying to get 100% code coverage is often a waste of time. You end up creating tests of very simple methods that don't really need testing (e.g., the getErrors() method that appears in later classes). You could be spending that time on better things. Testing experts say that having seventy or eighty percent code coverage is often good enough.

Still, measuring code coverage is worth doing, as we're about to see. If you add the test above and run the code coverage test again (remove the "ignore" directive if you tried it). You'll see that the Functions and Methods rating has risen from 50% to 75%. It was 50% because two of our four methods didn't pass. With the fix for the save() method, three of the four methods pass. The remaining problem is the get() method.

Navigate to the User1 class report if you're not already there. Scroll down and take a look at the get() method. The green lines are executing in our tests, the pink line (return '';) in the get() method is not.

This is valuable information. It tells us that we've never tested get() with an invalid field name. Create this new test and add it to the T1_UserConstructorTest.php file, just above the closing curly bracket:

public function testGetInvalid() {
    $user = new User1();

    $result = $user->get('not a field');
    assertEquals('', $result);

If you add the two extra tests and run the code coverage tests again, then drill down to the User1 class, you should see everything is now at 100% — full coverage.

If you look at the right-side of the Functions and Methods column, near the top you'll see the CRAP index. The CRAP index for the whole class is 8. About half of that is our constructor.

CRAP Index

When you drill down to look at the report for any individual class, you'll also get the CRAP index (sometimes written as C.R.A.P) for each class. This is not a coverage measure. CRAP stands for "Change Risk Anti-Patterns." It essentially measures the complexity of your code. The higher the index, the more difficult it is to maintain and improve, and the more likely it is that changes will break the code. Any CRAP index under 5 for a method is generally considered good, and some don't worry about the CRAP index until it approaches 30.

In production code, some classes have a very high CRAP index (sometimes into the hundreds or even thousands) because the complexity is necessary to perform the required actions of the class. It's often good to refactor code to reduce the CRAP index, but not always. Simple classes will have very low CRAP indexes, but the requirements of the task often force it higher.

The CRAP index of our constructor (8) is pushing things a little, and moving the validation out of the class entirely as we discussed a couple of articles ago, will definitely bring it back down.

When working with more complex classes, expect a higher index. Look at classes with a high index to see if simplifying them will improve the code, but don't try to reduce the CRAP index just for the sake of lowering the number, especially if doing so will remove important sanity checks or fail to meet the requirements of the task the class was written for.

Being overly concerned about the CRAP index can lead to a proliferation of very small classes and very small functions. That can lead to code that is very hard to maintain and sub-par performance.

Other Classes

Look at the other classes that don't have 100% ratings. Some of them will be improved by adding the two tests above. In others, you'll see that we didn't bother to test certain methods at all, or failed to test the methods under certain conditions. As an exercise, you can try to edit the other test classes to correct those problems, but remember that in the real world, obsessing about the coverage can be counter-productive.

Coming Up

In the next article, we'll look at integration testing, which tests components working together. with other components.

For more information on how to use MODX to create a web site, see my web site Bob's Guides, or better yet, buy my book: MODX: The Official Guide.

Looking for high-quality, MODX-friendly hosting? As of May 2016, Bob's Guides is hosted at A2 hosting. (More information in the box below.)

Comments (0)

Please login to comment.