Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

This page intentionally left blank

Chapter 5

Principles of Test

Automation

About This Chapter

Chapter 3, Goals of Test Automation, described the goals we should strive to

achieve to help us be successful at automating our unit tests and customer tests.

Chapter 4, Philosophy of Test Automation, discussed some of the differences in

the way people approach software design, construction, and testing. This pro-

vides the background for the principles that experienced test automaters follow

while automating their tests. I call them “principles” for two reasons: They are

too high level to be patterns and they represent a value system that not everyone

will share. A different value system may cause you to choose different patterns

than the ones presented in this book. Making this value system explicit will, I

hope, accelerate the process of understanding where we disagree and why.

The Principles

When Shaun Smith and I came up with the list in the original Test Automation

Manifesto [TAM], we considered what was driving us to write tests the way we

did. The Manifesto is a list of the qualities we would like to see in a test—not a

set of patterns that can be directly applied. However, those principles have led us

to identify a number of somewhat more concrete principles, some of which are

described in this chapter. What makes these principles different from the goals is

that there is more debate about them.

Principles are more “prescriptive” than patterns and higher level in nature. Un-

like patterns, they don’t have alternatives, but rather are presented in a “do this

because” fashion. To distinguish them from patterns, I have given them imperative

names rather than the noun-phrase names I use for goals, patterns, and smells.

For the most part, these principles apply equally well to unit tests and story-

tests. A possible exception is the principle Verify One Condition per Test, which

may not be practical for customer tests that exercise more involved chunks of

functionality. It is, however, still worth striving to follow these principles and to

deviate from them only when you are fully cognizant of the consequences.

Principle: Write the Tests First

Test-driven development is very much an acquired habit. Once one has “gotten

the hang of it,” writing code in any other way can seem just as strange as TDD

seems to those who have never done it. There are two major arguments in favor

of doing TDD:

1. The unit tests save us a lot of debugging effort—effort that often fully

offsets the cost of automating the tests.

2. Writing the tests before we write the code forces the code to be designed

for testability. We don’t need to think about testability as a separate

design condition; it just happens because we have written tests.

Principle: Design for Testability

Given the last principle, this principle may seem redundant. For developers

who choose to ignore Write the Tests First, Design for Testability becomes an

even more important principle because they won’t be able to write automated

tests after the fact if the testability wasn’t designed in. Anyone who has tried

to retroﬁ t automated unit tests onto legacy software can testify to the difﬁ culty

this raises. Mike Feathers talks about special techniques for introducing tests in

this case in [WEwLC].

Principle: Use the Front Door First

Objects have several kinds of interfaces. There is the “public” interface that clients

are expected to use. There may also be a “private” interface that only close friends

should use. Many objects also have an “outgoing interface” consisting of the used

part of the interfaces of any objects on which they depend.

The types of interfaces we use inﬂ uence the robustness of our tests. The use of

Back Door Manipulation (page 327) to set up the ﬁ xture or verify the expected

outcome or a test can result in Overcoupled Software (see Fragile Test on page

239) that needs more frequent test maintenance. Overuse of Behavior Veriﬁ ca-

tion (page 468) and Mock Objects (page 544) can result in Overspeciﬁ ed Software

(see Fragile Test) and tests that are more brittle and may discourage developers

from doing desirable refactorings.

Also known as:

Test-Driven

Development,

Test-First

Development

Chapter 5 Principles of Test Automation

Also known as:

Front Door First

When all choices are equally effective, we should use round-trip tests to test

our SUT. To do so, we test an object through its public interface and use State

Veriﬁ cation (page 462) to determine whether it behaved correctly. If this is not suf-

ﬁ cient to accurately describe the expected behavior, we can make our tests layer-

crossing tests and use Behavior Veriﬁ cation to verify the calls the SUT makes to

depended-on components (DOCs). If we must replace a slow or unavailable DOC

with a faster Test Double (page 522), using a Fake Object (page 551) is preferable

because it encodes fewer assumptions into the test (the only assumption is that the

component that the Fake Object replaces is actually needed).

Principle: Communicate Intent

Fully Automated Tests, especially Scripted Tests (page 285), are programs. They

need to be syntactically correct to compile and semantically correct to run success-

fully. They need to implement whatever detailed logic is required to put the SUT

into the appropriate starting state and to verify that the expected outcome has

occurred. While these characteristics are necessary, they are not sufﬁ cient because

they neglect the single most important interpreter of the tests: the test maintainer.

Tests that contain a lot of code

or Conditional Test Logic (page 200) are

usually Obscure Tests (page 186). They are much harder to understand because

we need to infer the “big picture” from all the details. This reverse engineering

of meaning takes extra time whenever we need to revisit the test either to main-

tain it or to use the Tests as Documentation. It also increases the cost of owner-

ship of the tests and reduces their return on investment.

Tests can be made easier to understand and maintain if we Communi-

cate Intent. We can do so by calling Test Utility Methods (page 599) with

Intent-Revealing Names [SBPP] to set up our test ﬁ xture and to verify that the

expected outcome has been realized. It should be readily apparent within the

Test Method (page 348) how the test ﬁ xture inﬂ uences the expected outcome of

each test—that is, which inputs result in which outputs. A rich library of Test

Utility Methods also makes tests easier to write because we don’t have to code

the details into every test.

Principle: Don’t Modify the SUT

Effective testing often requires us to replace a part of the application with a Test

Double or override part of its behavior using a Test-Speciﬁ c Subclass (page 579).

This may be because we need to gain control over its indirect inputs or because we

need to perform Behavior Veriﬁ cation by intercepting its indirect outputs. It may

Anything more than about ten lines is getting to be too much.

Also known as:

Higher-Level

Language,

Single-Glance

Readable

The Principles

also be because parts of the application’s behavior have unacceptable side effects or

dependencies that are impossible to satisfy in our development or test environment.

Modifying the SUT is a dangerous thing whether we are putting in Test

Hooks (page 709), overriding behavior in a Test-Speciﬁ c Subclass, or replacing

a DOC with a Test Double. In any of these circumstances, we may no longer

actually be testing the code we plan to put into production.

We need to ensure that we are testing the software in a conﬁ guration that is

truly representative of how it will be used in production. If we do need to replace

something the SUT depends on to get better control of the context surrounding the

SUT, we must make sure that we are doing so in a representative way. Otherwise,

we may end up replacing part of the SUT that we think we are testing. Suppose,

for example, that we are writing tests for objects X, Y, and Z, where object X

depends on object Y, which in turn depends on object Z. When writing tests for

X, it is reasonable to replace Y and Z with a Test Double. When testing Y, we can

replace Z with a Test Double. When testing Z, however, we cannot replace it with

a Test Double because Z is what we are testing! This consideration is particularly

salient when we have to refactor the code to improve its testability.

When we use a Test-Speciﬁ c Subclass to override part of the behavior of an

object to allow testing, we have to be careful that we override only those meth-

ods that the test speciﬁ cally needs to null out or use to inject indirect inputs. If

we choose to reuse a Test-Speciﬁ c Subclass created for another test, we must

ensure that it does not override any of the behavior that this test is verifying.

Another way of looking at this principle is as follows: The term SUT is rela-

tive to the tests we are writing. In our “X uses Y uses Z” example, the SUT for

some component tests might be the aggregate of X, Y, and Z; for unit testing

purposes, it might be just X for some tests, just Y for other tests, and just Z for

yet other tests. Just about the only time we consider the entire application to be

the SUT is when we are doing user acceptance testing using the user interface

and going all the way back to the database. Even here, we might be testing only

one module of the entire application (e.g., the “Customer Management Mod-

ule”). Thus “SUT” rarely equals “application.”

Principle: Keep Tests Independent

When doing manual testing, it is common practice to have long test procedures that

verify many aspects of the SUT’s behavior in a single test. This aggregation of tasks is

necessary because the steps involved in setting up the starting state of the system for

one test may simply repeat the steps used to verify other parts of its behavior. When

tests are executed manually, this repetition is not cost-effective. In addition, human

testers have the ability to recognize when a test failure should preclude continuing

Also known as:

Independent

Test

Chapter 5 Principles of Test Automation

execution of the test, when it should cause certain tests to be skipped, or when the

failure is immaterial to subsequent tests (though it may still count as a failed test.)

If tests are interdependent and (even worse) order dependent, we will deprive

ourselves of the useful feedback that individual test failures provide. Interacting

Tests (see Erratic Test on page 228) tend to fail in a group. The failure of a test

that moved the SUT into the state required by the dependent test will lead to

the failure of the dependent test, too. With both tests failing, how can we tell

whether the failure reﬂ ects a problem in code that both tests rely on in some

way or whether it signals a problem in code that only the ﬁ rst test relies on?

When both tests fail, we can’t tell. And we are talking about only two tests in

this case—imagine how much worse matters would be with tens or even hun-

dreds of Interacting Tests.

An Independent Test can be run by itself. It sets up its own Fresh Fix-

ture (page 311) to put the SUT into a state that lets it verify the behavior it is

testing. Tests that build a Fresh Fixture are much more likely to be independent

than tests that use a Shared Fixture (page 317). The latter can lead to various

kinds of Erratic Tests, including Lonely Tests, Interacting Tests, and Test Run

Wars. With independent tests, unit test failures give us Defect Localization to

help us pinpoint the source of the failure.

Principle: Isolate the SUT

Some pieces of software depend on nothing but the (presumably correct) run-

time system or operating system. Most pieces of software build on other pieces

of software developed by us or by others. When our software depends on other

software that may change over time, our tests may suddenly start failing because

the behavior of the other software has changed. This problem, which is called

Context Sensitivity (see Fragile Test), is a form of Fragile Test.

When our software depends on other software whose behavior we cannot

control, we may ﬁ nd it difﬁ cult to verify that our software behaves properly

with all possible return values. This is likely to lead to Untested Code (see Pro-

duction Bugs on page 268) or Untested Requirements (see Production Bugs).

To avoid this problem, we need to be able to inject all possible reactions of the

DOC into our software under the complete control of our tests.

Whatever application, component, class, or method we are testing, we should

strive to isolate it as much as possible from all other parts of the software that

we choose not to test. This isolation of elements allows us to Test Concerns

Separately and allows us to Keep Tests Independent of one another. It also helps

us create a Robust Test by reducing the likelihood of Context Sensitivity caused

by too much coupling between our SUT and the software that surrounds it.

The Principles

We can satisfy this principle by designing our software such that each piece

of depended-on software can be replaced with a Test Double using Dependency

Injection (page 678) or Dependency Lookup (page 686) or overridden with a

Test-Speciﬁ c Subclass that gives us control of the indirect inputs of the SUT.

This design for testability makes our tests more repeatable and robust.

Principle: Minimize Test Overlap

Most applications have lots of functionality to verify. Proving that all of the

functionality works correctly in all possible combinations and interaction sce-

narios is nearly impossible. Therefore, picking the tests to write is an exercise in

risk management.

We should structure our tests so that as few tests as possible depend on a

particular piece of functionality. This may seem counter-intuitive at ﬁ rst be-

cause one would think that we would want to improve test coverage by testing

the software as often as possible. Unfortunately, tests that verify the same func-

tionality typically fail at the same time. They also tend to need the same mainte-

nance when the functionality of the SUT is modiﬁ ed. Having several tests verify

the same functionality is likely to increase test maintenance costs and probably

won’t improve quality very much.

We do want to ensure that all test conditions are covered by the tests that we

do use. Each test condition should be covered by exactly one test—no more, no

less. If it seems to provide value to test the code in several different ways, we

may have identiﬁ ed several different test conditions.

Principle: Minimize Untestable Code

Some kinds of code are difﬁ cult to test using Fully Automated Tests. GUI com-

ponents, multithreaded code, and Test Methods immediately spring to mind as

“untestable” code. All of these kinds of code share the same problem: They are

embedded in a context that makes it hard to instantiate or interact with them

from automated tests.

Untestable code simply won’t have any Fully Automated Tests to protect it

from those nefarious little bugs that can creep into code when we aren’t look-

ing. That makes it more difﬁ cult to refactor this code safely and more danger-

ous to modify existing functionality or introduce new functionality.

It is highly desirable to minimize the amount of untestable code that we have

to maintain. We can refactor the untestable code to improve its testability by

moving the logic we want to test out of the class that is causing the lack of test-

ability. For active objects and multithreaded code, we can refactor to Humble

Executable (see Humble Object on page 695). For user interface objects, we

Chapter 5 Principles of Test Automation

can refactor to Humble Dialog (see Humble Object). Even Test Methods can

have much of their untestable code extracted into Test Utility Methods, which

can then be tested.

When we Minimize Untestable Code, we improve the overall test coverage of

our code. In so doing, we also improve our conﬁ dence in the code and extend

our ability to refactor at will. The fact that this technique improves the quality

of the code is yet another beneﬁ t.

Principle: Keep Test Logic Out of Production Code

When the production code hasn’t been designed for testability (whether as a

result of test-driven development or otherwise), we may be tempted to put

“hooks” into the production code to make it easier to test. These hooks typi-

cally take the form of

if testing then ... and may either run alternative logic or

prevent certain logic from running.

Testing is about verifying the behavior of a system. If the system behaves dif-

ferently when under test, then how can we be certain that the production code

actually works? Even worse, the test hooks could cause the software to fail in

production!

The production code should not contain any conditional statements of the if

testing then

sort. Likewise, it should not contain any test logic. A well-designed

system (from a testing perspective) is one that allows for the isolation of func-

tionality. Object-oriented systems are particularly amenable to testing because

they are composed of discrete objects. Unfortunately, even object-oriented sys-

tems can be built in such a way as to be difﬁ cult to test, and we may still en-

counter code with embedded test logic.

Principle: Verify One Condition per Test

Many tests require a starting state other than the default state of the SUT, and

many operations of the SUT leave it in a different state from its original state.

There is a strong temptation to reuse the end state of one test condition as the

starting state of the next test condition by combining the veriﬁ cation of the two

test conditions into a single Test Method because this makes testing more efﬁ -

cient. This approach is not recommended, however, because when one assertion

fails, the rest of the test will not be executed. As a consequence, it becomes more

difﬁ cult to achieve Defect Localization.

Verifying multiple conditions in a single test makes sense when we execute

tests manually because of the high overhead of test setup and because the live-

ware can adapt to test failures. It is too much work to set up the ﬁ xture for a

large number of manual tests, so human testers naturally tend to write long

Also known as:

No Test Logic

in Production

Code

The Principles

Also known as:

Single-Condition

Test

multiple-condition tests.

They also have the intelligence to work around any

issues they encounter so that all is not lost if a single step fails. In contrast, with

automated tests, a single failed assertion will cause the test to stop running and

the rest of the test will provide no data on what works and what doesn’t.

Each Scripted Test should verify a single test condition. This single-mindedness

is possible because the test ﬁ xture is set up programmatically rather than by a

human. Programs can set up ﬁ xtures very quickly and they don’t have trouble ex-

ecuting exactly the same sequence of steps hundreds of times! If several tests need

the same test ﬁ xture, either we can move the Test Methods into a single Testcase

Class per Fixture (page 631) so we can use Implicit Setup (page 424) or we can

call Test Utility Methods to set up the ﬁ xture using Delegated Setup (page 411).

We design each test to have four distinct phases (see Four-Phase Test on

page 358) that are executed in sequence: ﬁ xture setup, exercise SUT, result

veriﬁ cation, and ﬁ xture teardown.

• In the ﬁ rst phase, we set up the test ﬁ xture (the “before” picture) that

is required for the SUT to exhibit the expected behavior as well as any-

thing we need to put in place to observe the actual outcome (such as

using a Test Double).

• In the second phase, we interact with the SUT to exercise whatever

behavior we are trying to verify. This should be a single, distinct behav-

ior; if we try to exercise several parts of the SUT, we are not writing a

Single-Condition Test.

• In the third phase, we do whatever is necessary to determine whether

the expected outcome has been obtained and fail the test if it has not.

• In the fourth phase, we tear down the test ﬁ xture and put the world

back into the state in which we found it.

Note that there is a single exercise SUT phase and a single result veriﬁ cation

phase. We avoid having a series of such alternating calls (exercise, verify, exercise,

verify) because that approach would be trying to verify several distinct condi-

tions—something that is better handled via distinct Test Methods.

One possibly contentious aspect of Verify One Condition per Test is what

we mean by “one condition.” Some test drivers insist on one assertion per test.

This insistence may be based on using a Testcase Class per Fixture organization

of the Test Methods and naming each test based on what the one assertion is

Clever testers often use automated test scripts to put the SUT into the correct starting

state for their manual tests, thereby avoiding long manual test scripts.

Chapter 5 Principles of Test Automation

verifying.

Having one assertion per test makes such naming very easy but also

leads to many more test methods if we have to assert on many output ﬁ elds. Of

course, we can often comply with this interpretation by extracting a Custom

Assertion (page 474) or Veriﬁ cation Method (see Custom Assertion) that allows

us to reduce the multiple assertion method calls to a single call. Sometimes that

approach makes the test more readable. When it doesn’t, I wouldn’t be too dog-

matic about insisting on a single assertion.

Principle: Test Concerns Separately

The behavior of a complex application consists of the aggregate of a large num-

ber of smaller behaviors. Sometimes several of these behaviors are provided by

the same component. Each of these behaviors is a different concern and may

have a signiﬁ cant number of scenarios in which it needs to be veriﬁ ed.

The problem with testing several concerns in a single Test Method is that

this method will be broken whenever any of the tested concerns is modiﬁ ed.

Even worse, it won’t be obvious which concern is the one at fault. Identify-

ing the real culprit typically requires Manual Debugging (see Frequent Debug-

ging on page 248) because of the lack of Defect Localization. The net effect is

that more tests will fail and each test will take longer to troubleshoot and ﬁ x.

Refactoring is also made more difﬁ cult by testing several concerns in the same

test; it will be harder to “tease apart” the eager class into several independent

classes, each of which implements a single concern, because the tests will need

extensive redesign.

Testing our concerns separately allows a failure to tell us that we have a

problem in a speciﬁ c part of our system rather than simply saying that we

have a problem somewhere. This approach to testing also makes it easier

to understand the behavior now and to separate the concerns in subsequent

refactorings. That is, we should just be able to move a subset of the tests to

a different Testcase Class (page 373) that veriﬁ es the newly created class; it

shouldn’t be necessary to modify the test much more than changing the class

name of the SUT.

Principle: Ensure Commensurate Effort and Responsibility

The amount of effort it takes to write or modify tests should not exceed the

effort it takes to implement the corresponding functionality. Likewise, the tools

required to write or maintain the test should require no more expertise than the

tools used to implement the functionality. For example, if we can conﬁ gure the

For example, AwaitingApprovalFlight.validApproverRequestShouldBeApproved.

The Principles