Unit, Integration, Functional, and Acceptance Testing

Defining the most common kinds of software tests.


In the last article, we discussed software testing and why you should do it. In this one, we'll look at the definitions of four different kinds of testing: unit testing, integration testing, functional testingi, and acceptance testing. In later articles in the series, we'll explore doing the actual testing using those methods.

Our coverage of testing methods here will be fairly shallow, since we're mainly just looking at the definitions of unit, integration, functional, and acceptance testing. We'll go into more detail in later articles when we create some actual tests. In the following article, we'll cover installing and using the tools necessary to do the testing.


MODX logo

Overview

If you read web articles on software testing, you'll see two things. First, it's a huge area with dozens of topics (there's a fairly exhaustive list of testing types here), some of them quite advanced (e.g., Requirement Traceability Matrices). Second, not everyone uses the same terms or organizes them the same way. This article is an introduction and will present the terms as I use them. Be aware that there will be people who disagree with almost any definition of the testing terms here.

Another thing you'll notice after looking at articles on software testing is that they all use fairly simple examples (as we will do here). One reason for this is that they are easier to understand. Another, though, is that in the real world of software development, tests don't always fall neatly into preset categories. A method to get a password hash is easy to understand and easy to write a unit test for. So is a simple calculator object that adds, subtracts, multiplies, and divides. A class that aggregates resources for display, like getResources or pdoResources, is not at all simple. Performance constraints can mean the code is very interdependent and difficult to create unit tests for. (In other words, making the code modular and easily testable would slow it down too much.) Take any "rules" about testing with a grain of salt.

Whatever you're testing, a class, a method, an API, a website, or some combination of these is called the "System Under Test" (SUT). It's an ugly term that I don't much care for, and won't use in this series, but you should be aware of it if you read other articles on software testing.

To start things off, let's look at unit testing.


Unit Testing

You may already be familiar with unit tests, since they've been around the longest. You don't have to, but it's a lot easier and more reliable to use PhpUnit or some other framework to perform them. Unit tests test the smallest possible units of your system, usually the individual methods of a class or the functions contained in a section of code. Imagine that you have a class named Calculator that contains a method called add(). Here's a simple unit test for that method using PhpUnit:

<?php
$c = new Calculator();

$result = $c->add(1,2);
assertEquals(3,$result);

$result = $c->add(1.2, 2.2);
assertEquals(3.3, $result);

Obviously, a real unit test of your calculator would include many more assertions. You'd want to test with very large numbers, zero, negative numbers, and possibly numbers using scientific notation.


Assertions

Assertions are at the heart of unit testing. They are built into the PhpUnit framework. Here are some commonly used assertions. The names are fairly straightforward:

  • assertEquals()
  • assertNotEquals()
  • assertTrue()
  • assertFalse()
  • assertEmpty()
  • assertGreaterThan()
  • assertLessThan()
  • assertFileExists()
  • assertArrayHasKey()
  • assertInstanceof()
  • assertContains()

You can see a more complete list in the left sidebar here.

In a typical unit test, the assertions would be inside a test class that extended the PhpUnitTestCase class. By default, they would be in one of these two forms:

$this->assertEquals(3.3, $result);
self::assertEquals(3.3, $result);

I used the short forms in my example above to make it more readable. Actually, you can use the short versions I used if you (manually) 'include' the src/Framework/Assert/Functions.php sourcecode file that comes with PHPUnit. Which form you use is a matter of personal preference. Using the short version is slightly slower, since it makes and extra call to the wrapper functions, but the difference is very small and using the wrapper functions saves you a lot of typing.

All assertions have an optional final argument that's a string containing a message that will be displayed if the assertion fails:

assertEquals(3.3, $result, "3.3 does not equal " . $result);

In the code above, the first argument is the expected value, the second one is the actual value, and the third is the error message to display if the assertion fails. In unit testing assertions, the expected value always comes first.

Often, the error message string is unnecessary, since PhpUnit will print a fairly sensible error message on its own. The message is sometimes necessary, though, if the assertion is more complex, as in this example:

assertTrue($result < 25 || $result > 999,
    "Result " . $result . " is out of range");

External Dependencies

What if the method you're testing calls one or more other methods to do its job? It's a principle of unit testing that you only test one thing per test and do not test external dependencies. You'd test those other methods in another test. In order to make the test work, though, you need to create fake versions of those other methods. The general term for these fake versions is "test doubles." Test doubles include things called dummies, fakes, stubs, mocks, and spies. Unfortunately, there is considerable controversy on what those terms mean. Some people say all mocks are stubs, others say all stubs are mocks. Some people refer to all test doubles as mocks.

Here's a short version of how I use those terms:

  • Dummy — A parameter that has no effect on your test, but is required by the method you're testing
  • Fake — A simulation of a real class that takes some shortcuts to do it's work (not fit for production use) — usually a subclass of the class being tested
  • Stub — A test double that returns a hard-coded value — used in place of an external method
  • Mock — A smarter test double you create for an external method that can verify its input, how many times its called, and/or what methods it calls.
  • Spy — An even smarter test double that records its interactions with various external methods.

I tend to avoid spies, since they're more difficult to implement and tend to make tests fail that shouldn't when you modify your code. Dummies, are obvious. Imagine that you're testing a method that validates a user, and that method requires a $rememberMe parameter that's not relevant to your current test. The variable would determine if another method, called rememberUser() was called. You might create a dummy &rememberMe variable to satisfy the parameter requirements. That variable would have no effect on the test.

You might also create a mock version of rememberUser() that makes sure that it's called, but only called once when the $rememberMe parameter is true and is not called when that parameter is false. In this case, the $rememberMeparameter is no longer a true dummy, since its value affects the test.

The only reason for knowing these terms is that both PhpUnit and Codeception have built-in methods for implementing stubs and mocks. PhpUnit calls both of them mocks. Codeception (which is essentially a wrapper that extends PhpUnit) distinguishes between them, though at this writing, the difference between them is not at all clear from the documentation. In any case, if you define a stub or mock the right way, it will be called instead of the "real" method when the method or class you're testing calls it.

In my opinion, it's not critical to have a firm grasp of the differences between these terms as long as you can create tests that work properly to test your code. Not everyone would agree.

We'll look at specific examples of these various test doubles in much more detail in future articles in this series, once we've installed some testing software.


Integration Testing

In unit testing, you test small parts of your code in isolation, but two parts of the code that work when tested individually may not work together. In Integration testing, you would test them together. There are different strategies for integration testing. You can start by testing pairs of methods that work together (e.g. user validation and user creation). You can gradually create larger and large larger subsystems of units for testing. You can also test the entire system as a whole from the beginning. The strategy you use will depend on the size of your project and your personal preferences.

It's possible to do integration testing with PhpUnit by creating new tests that don't use stubs and mocks. In other words, they call the actual classes and methods. This is acceptable, in my opinion, as long as real unit tests with stubs and mocks exist to test the individual classes and methods, and the integration tests are not performed on a production server.


Functional Testing

While unit testing tests individual units of your code and integration testing makes sure those units can work together, a functional test makes sure that parts of your project function properly *for the user*. Unit tests and integration tests take the perspective of the programmer, a functional test takes the perspective of the user. Say we're testing a car. Unit tests would tell us if the alternator is producing electricity and the fuel injection system is creating the correct air/fuel mixture in various conditions. An integration test could check to see if the alternator is keeping the battery charged in various conditions and if the braking system is turning on the brake lights. A functional test might check to make sure that a driver can reach the gas pedal and that the car accelerates properly when the pedal is pressed and stops accelerating when it's released.

With a web site, a unit test will tell us whether a user with the correct credentials is validated by the validation method and a user with bad credentials is not. An integration test will tell us if the user registration system will successfully create a user in the database (given a valid set of user data), and fail gracefully if the user data is invalid. A Functional test might tell us if a user with the correct credentials can log in successfully and be redirected to the proper page.

Functional testing, as defined in the Codeception platform, simulates a visit by the user, but without creating a display, unlike acceptance testing, which simulates user actions by (usually) using a real browser display, entering text, and clicking on links and buttons.

Codeception provides frameworks for automating functional testing. Unfortunately, it doesn't provide support for the MODX platform yet (it does support Symfony, Facebook, Yii, Laravel, and Zend). If it did, it would be possible to do functional testing in MODX without involving a web browser.

In spite of the lack of support, it's still possible to create many functional tests in MODX by instantiating MODX and using some of its built-in components as part of your tests.

It's possible to instantiate MODX, then instantiate a processor class. Once you've done that, you can pass the processor class a variety of inputs, test what the processor returns, and make sure that it has performed the appropriate actions. For example, you might test the resource/create processor to make sure that it returns an appropriate error if the data it gets is invalid, and creates a resource in the database when all the proper fields are set and valid.

Functional tests run much faster than acceptance tests, but tend to be more finicky and more difficult to create. The effort to create them for MODX is sometimes not worth the trouble (especially when they run in the MODX Manager as CMPs do). Acceptance tests can test the same things, are easier to create, and tend to be a better measure of whether the software will work for the user, though they take longer to run.

Another way to think of functional testing is to read the section below on acceptance testing, then imagine doing it without involving a browser. This would involve having a piece of middleware that simulated all the things a browser lets you do, but without creating a display.


Acceptance Testing

Imagine that you could build a robot that would log in to an actual MODX site, or the MODX Manager, perform a series of actions, make sure that everything worked as expected, and report any failures. This is essentially what Codeception acceptance tests do. You basically write a script for the tester to follow and run it.

For example, you might write a Manager login test that navigates to the login page, enters the credentials, hits submit, then checks to see that it's been forwarded to the Manager dashboard. The Codeception scripts are quite readable and part of that test might look like this:

class Login {

    /* Set some variables *
    public static $managerUrl = 'manager/';
    public static $usernameField = '#modx-login-username';
    public static $passwordField = '#modx-login-password';
    public static $username = 'JoeTester';
    public static $password = 'testerPassword';

    /* Identify the Login button by its ID */
    public static $loginButton = '#modx-login-btn';

    /* Tell the testing framework what kind of User
       we're implementing */
    /** @var \AcceptanceTester $tester */
    protected $tester;

    /* Create the test user */
    public function __construct(\AcceptanceTester $I) {
        $this->tester = $I;
    }

    /* @throws Exception */
    public function login($username, $password) {
        /* @var \AcceptanceTester $I */

        $I = $this->tester;
        $I->amOnPage(self::$managerUrl);
        $I->fillField(self::$usernameField, $username);
        $I->fillField(self::$passwordField, $password);
        $I->click(self::$loginButton);
        $I->see('Welcome');
    }
}

Note that this is not a real acceptance test (which would be in it's own class), and it would not run on its own. This is a "helper" function I wrote to work in various real acceptance tests. It simply logs in the test user. It shows how easy it is to understand the code of an acceptance test. A real test might call this function, then perform these tests to look at what's on the screen to make sure we've reached the MODX Manager:

$I->see("MODX Revolution");
$I->see("Content");
$I->see("Media");
$I->see("Manage");

Other tests could go on to perform various operations in MODX (clicking on links and buttons, filling forms, etc.) and make sure they all work as they should. A different acceptance test could test what happens when the user's credentials are bad. The sequence of actions performed is called a "scenario". There are tools that let you write your scenario in (almost) plain English (e.g. Gherkin), which is then translated into acceptance test code, but they are beyond the scope of these articles and I prefer not to use them because they introduce a whole other level of operations between you and your tests. We'll see a real acceptance test when we get to the article on acceptance testing.


Coming Up

In the next few articles, we'll see how to install and configure Composer, Codeception, and PhpUnit. Once they're installed, we'll look at how to do some real tests in later articles.



For more information on how to use MODX to create a web site, see my web site Bob's Guides, or better yet, buy my book: MODX: The Official Guide.

Looking for high-quality, MODX-friendly hosting? As of May 2016, Bob's Guides is hosted at A2 hosting. (More information in the box below.)



Comments (0)


Please login to comment.

  (Login)