Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

Chapter 6 Test Automation Strategy

The xUnit Sweet Spot

The xUnit family works best when we can organize our tests as a large set of

small tests, each of which requires a small test ﬁ xture that is relatively easy to set

up. This allows us to create a separate test for each test scenario of each object.

The test ﬁ xture should be managed using a Fresh Fixture (page 311) strategy by

setting up a new Minimal Fixture (page 302) for each test.

xUnit works best when we write tests against software APIs and then test

single classes or small groups of classes in isolation. This approach allows us to

build small test ﬁ xtures that can be instantiated quickly.

When doing customer tests, xUnit works best if we deﬁ ne a Higher-Level

Language (page 41) with which to describe our tests. This choice moves the

level of abstraction higher, away from the nitty-gritty of the technology and

closer to the business concepts that customers understand. From here, it is a

very small step to convert these tests to Data-Driven Tests (page 288) imple-

mented in xUnit or Fit.

Note that many of the higher-level patterns and principles described in this

book apply equally well to both Fit tests and xUnit tests. I have also found them

to be useful when working with commercial GUI-based testing tools, which

typically use a “record and playback” metaphor. The ﬁ xture management

patterns are particularly salient in this arena, as are reusable “test components”

that may be strung together to form a variety of test scripts. This is entirely

analogous to the xUnit practice of single-purpose Test Methods (page 348)

calling reusable Test Utility Methods (page 599) to reduce their coupling to the

SUT’s API.

Which Test Fixture Strategy Do We Use?

The test ﬁ xture management strategy is strategic because it has a large impact on

the execution time and robustness of the tests. The effects of picking the wrong

strategy won’t be felt immediately because it takes at least a few hundred tests

before the Slow Tests (page 253) smell becomes evident and probably several

months of development before the High Test Maintenance Cost smell starts to

emerge. Once these smells appear, however, the need to change the test automa-

tion strategy will become apparent—and its cost will be signiﬁ cant because of

the number of tests affected.

What Is a Fixture?

Every test consists of four parts, as described in Four-Phase Test (page 358). In

the ﬁ rst phase, we create the SUT and everything it depends on and put them

into the state required to exercise the SUT. In xUnit, we call everything we need

in place to exercise the SUT the test ﬁ xture, and we call the part of the test logic

that we execute to set it up the ﬁ xture setup phase of the test.

At this point, a word of caution is in order. The term “ﬁ xture” means many

things to many people:

• Some variants of xUnit keep the concept of the ﬁ xture separate from

the Testcase Class (page 373) that creates it. JUnit and its direct ports

fall into this category.

• Other members of the xUnit family assume that an instance of the Test-

case Class “is a” ﬁ xture. NUnit is a good example.

• A third camp uses an entirely different name for the ﬁ xture. For

example, RSpec captures the pre-conditions of the test in a test con-

text class that holds the Test Methods (same idea as NUnit but with

different terminology).

• The term “ﬁ xture” is used to mean entirely different things in other

kinds of test automation. In Fit, for example, it means the custom-built

parts of the Data-Driven Test Interpreter [GOF] that we use to deﬁ ne

our Higher-Level Language.

The “class ‘is a’ ﬁ xture” approach assumes the Testcase Class per Fixture (page 631)

approach to organizing the tests. When we choose a different way of organizing

the tests, such as Testcase Class per Class (page 617) or Testcase Class per Fea-

ture (page 624), this merging of the concepts of test ﬁ xture and Testcase Class can

be confusing. Throughout this book, I use “test ﬁ xture”—or just “ﬁ xture”—to

mean “the pre-conditions of the test” and Testcase Class to mean “the class that

contains the Test Methods and any code needed to set up the test ﬁ xture.”

The most common way to set up the ﬁ xture is to use front door ﬁ xture setup

by calling the appropriate methods on the SUT to construct the objects. When

the state of the SUT is stored in other objects or components, we can do Back

Door Setup (see Back Door Manipulation on page 327) by inserting the neces-

sary records directly into the other component on which the behavior of the

SUT depends. We use Back Door Setup most often with databases or when

we need to use a Mock Object (page 544) or Test Double (page 522); these

concepts are covered in more detail in Chapter 13, Testing with Databases, and

Chapter 11, Using Test Doubles.

Which Test Fixture Strategy Do We Use?

Chapter 6 Test Automation Strategy

Major Fixture Strategies

There are probably many ways to classify just about anything. For the purposes

of this discussion, we will classify our test ﬁ xture strategies based on what kinds

of test development work we need to do for each one.

The ﬁ rst and simplest ﬁ xture management strategy requires us to worry only

how we will organize the code to build the ﬁ xture for each test. That is, do we

put this code in our Test Methods, factor it into Test Utility Methods that we

call from our Test Methods, or put it into a setUp method in our Testcase Class?

This strategy involves the use of Transient Fresh Fixtures (see Fresh Fixture).

These ﬁ xtures live only in memory and very conveniently disappear as soon as

we are done with them.

A second strategy involves the use of Fresh Fixtures that, for one reason or

another, persist beyond the single Test Method that uses it. To keep them from

turning into Shared Fixtures (page 317), these Persistent Fresh Fixtures (see

Fresh Fixture) require explicit code to tear them down at the end of each test.

This requirement brings into play the ﬁ xture teardown patterns.

A third strategy involves persistent ﬁ xtures that are deliberately reused across

many tests. This Shared Fixture strategy is often used to improve the execu-

tion speed of tests that use a Persistent Fresh Fixture but comes with a fair

amount of baggage. These tests require the use of one of the ﬁ xture construc-

tion and teardown triggering patterns. They also involve tests that interact with

one another, whether by design or by consequence, which often leads to Erratic

Tests (page 228) and High Test Maintenance Costs.

Table 6.1 summarizes the ﬁ xture management overhead associated with each

of the three styles of ﬁ xtures.

Table 6.1 A Summary of the Fixture Setup and Teardown Requirements of the

Various Test Fixture Strategies

Set Up Code Tear Down Code Setup/Teardown

Triggering

Transient Fresh Fixture Yes

Persistent Fresh Fixture Yes Yes

Shared Fixture Yes Yes Yes

Note: The

Shared Fixture row assumes we are building a new Shared Fixture each test

run rather than using a Prebuilt Fixture (page 429).

Figure 6.4 illustrates the interaction between our goals, freshness of ﬁ xtures or

ﬁ xture reuse, and ﬁ xture persistence. It also illustrates a few variations of the

Shared Fixture.

Figure 6.4 A summary of the main test ﬁ xture strategies. Fresh Fixtures can be

either transient or persistent; Shared Fixtures must be persistent. An Immutable

Shared Fixture (see Shared Fixture) must not be modiﬁ ed by any test. As a

consequence, most tests augment the Shared Fixture with a Fresh Fixture that

they can modify.

The relationship between persistence and freshness is reasonably obvious for

two of these combinations. The persistent Fresh Fixture is discussed in more

detail later in this chapter. The transient Shared Fixture is inherently transient—

how we hold references to these ﬁ xtures is what makes them persist. Other than

this distinction, transient Shared Fixtures can be treated exactly like persistent

Shared Fixtures.

Transient Fresh Fixtures

In this approach, each test creates a temporary Fresh Fixture as it runs. Any

objects or records it requires are created by the test itself (though not necessarily

inside the Test Method). Because the test ﬁ xture visibility is restricted to the one

test alone, we ensure that each test is completely independent because it cannot

depend, either accidentally or on purpose, on the output of any other tests that

use the same ﬁ xture.

Transient

Persistent

Immutable

Shared

Fixture

Shared

Fixture

Fresh

Fixture

Transient

Persistent

Immutable

Shared

Fixture

Shared

Fixture

Fresh

Fixture

Which Test Fixture Strategy Do We Use?

Chapter 6 Test Automation Strategy

We call this approach Fresh Fixture because each test starts with a clean slate

and builds from there. It does not “inherit” or “reuse” any part of the ﬁ xture

from other tests or from a Prebuilt Fixture (page 429). Every object or record

used by the SUT is “fresh,” “brand new,” and not “previously enjoyed.”

The main disadvantage of using the Fresh Fixture approach is the additional

CPU cycles it takes to create all the objects for each test. As a consequence, the

tests may run more slowly than under a Shared Fixture approach, especially if

we use a Persistent Fresh Fixture.

Persistent Fresh Fixtures

A Persistent Fresh Fixture sounds a bit oxymoronic. We want the ﬁ xture to be

fresh, yet it persists beyond the lifetime of a single test! What kind of strategy is

that? Some might say “stupid,” but sometimes one has to do this.

We are “forced” into this strategy when we are testing components that are

tightly coupled to a database or other persistence mechanism. The obvious so-

lution is that we should not let the coupling be so tight, but rather make the

database a substitutable dependency of the component we are testing. This step

may not be practical when testing legacy software, however—yet we may still

want to partake of the beneﬁ ts of a Fresh Fixture. Hence the existence of the

Persistent Fresh Fixture strategy. The key difference between this strategy and the

Transient Fresh Fixture is the need for code to tear down the ﬁ xture after each test.

Persistent Fresh Fixtures can result in Slow Tests if the persistence of the ﬁ xture is

caused by the use of a database, ﬁ le system, or other high-latency dependency.

We can at least partially address the resulting Slow Tests by applying one or

more of the following patterns:

1. Construct a Minimal Fixture (the smallest ﬁ xture possible).

2. Speed up the construction by using a Test Double to replace the pro-

vider of any data that takes too long to set up.

3. If the tests still are not fast enough, minimize the size of the part of

the ﬁ xture we need to destroy and reconstruct each time by using an

Immutable Shared Fixture for any objects that are referenced but not

modiﬁ ed.

The project teams with which I have worked have found that, on average, our

tests run 50 times faster (yes, they take 2% as long) when we use Dependency

Injection (page 678) or Dependency Lookup (page 686) to replace the entire

database with a Fake Database (see Fake Object on page 551) that uses a set of

hash tables instead of tables. Each test may require many, many database opera-

tions to set up and tear down the ﬁ xture required by a single query in the SUT.

There is a lot to be said for minimizing the size and complexity of the test

ﬁ xture. A Minimal Fixture (see Minimal Fixture) is much easier to understand

and helps highlight the cause–effect relationship between the ﬁ xture and the ex-

pected outcome. In this regard, it is a major enabler of Tests as Documentation

(page 23). In some cases, we can make the test ﬁ xture much smaller by using

Entity Chain Snipping (see Test Stub on page 529) to eliminate the need to in-

stantiate those objects on which our test depends only indirectly. This tactic will

certainly speed up the instantiation of our test ﬁ xture.

Shared Fixture Strategies

Sometimes we cannot—or choose not to—use a Fresh Fixture strategy. In these

cases, we can use a Shared Fixture. In this approach, many tests reuse the same

instance of a test ﬁ xture.

The major advantage of Shared Fixtures is that we save a lot of execution

time in setting up and tearing down the ﬁ xture. The main disadvantage is con-

veyed by one of its aliases, Stale Fixture, and by the test smell that describes its

most common side effects, Interacting Tests (see Erratic Test). Although Shared

Fixtures do have other beneﬁ ts, most can be realized by applying other patterns

to Fresh Fixtures; Standard Fixture (page 305) avoids the ﬁ xture design and

coding effort for every test without actually sharing the ﬁ xture.

Now, if Shared Fixtures are so bad, why even discuss them? Because every-

one seems to go down this road at least once in his or her career—so we might

as well share the best available information about them should you venture

down that path. Mind you, this discussion isn’t meant to encourage anyone to

go down this path unnecessarily because it is paved with broken glass, infested

with poisonous snakes, and . . . well, you get my drift.

Given that we have decided to use a Shared Fixture (we did investigate every

possible alternative, didn’t we?), what are our options? We can make the fol-

lowing adjustments (Figure 6.5):

• How far and wide we share a ﬁ xture (e.g., a Testcase Class, all tests in

a test suite, all test run by a particular user)

• How often we recreate the ﬁ xture

Which Test Fixture Strategy Do We Use?

Chapter 6 Test Automation Strategy

Figure 6.5 The various ways we can manage a Shared Fixture. The strategies

are ordered by the length of the ﬁ xture’s lifetime, with the longestlasting ﬁ xture

appearing on the left.

The more tests that share a ﬁ xture, the more likely that one of them will make a

mess of things and spoil everything for all the tests that follow it. The less often

we reconstruct the ﬁ xture, the longer the effects of a messed-up ﬁ xture will per-

sist. For example, a Prebuilt Fixture can be set up outside the test run, thereby

avoiding the entire cost of setting up the ﬁ xture as part of the test run; unfor-

tunately, it can also result in Unrepeatable Tests (see Erratic Test) if tests don’t

clean up after themselves properly. This strategy is most commonly used with a

Database Sandbox (page 650) that is initialized using a database script; once the

ﬁ xture is corrupted, it must be reinitialized by rerunning the script. If the Shared

Fixture is accessible to more than one Test Runner (page 377), we may end up

in a Test Run War (see Erratic Test), in which tests fail randomly as they try to

use the same ﬁ xture resource at the same time as some other test.

We can avoid both Unrepeatable Tests and Test Run Wars by setting up the

ﬁ xture each time the test suite is run. xUnit provides several ways to do so,

including Lazy Setup (page 435), Suite Fixture Setup (page 441), and Setup

Decorator (page 447). The concept of “lazy initialization” should be familiar to

most object-oriented developers; here we just apply the concept to the construc-

tion of the test ﬁ xture. The latter two choices provide a way to tear down the

test ﬁ xture when the test run is ﬁ nished because they call a setUp method and a

corresponding tearDown at the appropriate times; Lazy Setup does not give us a

way to do this.

Chained Tests represent another option for setting up a Shared Fixture, one that

involves running the tests in a predeﬁ ned order and letting each test use the previ

ous test’s results as its test ﬁ xture. Unfortunately, once one test fails, many of the

tests that follow will provide erratic results because their pre-conditions have not

Shared Fixture Setup

Shared

Fixture

Setup

Decorator

Suite

Fixture

Setup

Lazy

Setup

Prebuilt

Fixture

Chained

Tests

Shared Fixture Setup

Shared

Fixture

Setup

Decorator

Suite

Fixture

Setup

Lazy

Setup

Prebuilt

Fixture

Chained

Tests

been satisﬁ ed. This problem can be made easier to diagnose by having each test

use Guard Assertions (page 490) to verify that its pre-conditions have been met.

As mentioned earlier, an Immutable Shared Fixture is a strategy for speeding

up tests that use a Fresh Fixture. We can also use an Immutable Shared Fixture

to make tests based on a Shared Fixture less erratic by restricting changes to a

smaller, mutable part of a Shared Fixture.

How Do We Ensure Testability?

The last strategic concern touched on in this chapter is ensuring testability. The

discussion here isn’t intended to be a complete treatment of the topic—it is too

large to cover in a single chapter on test strategy. Nevertheless, we shouldn’t

sweep this issue under the carpet either, because it deﬁ nitely has a major impact

on test automation. But ﬁ rst, I must climb onto my soapbox for a short digres-

sion into the development process.

Test Last—at Your Peril

Anyone who has tried to retroﬁ t unit tests onto an existing application has prob-

ably experienced a lot of pain! This is the hardest kind of test automation we

can do as well as the least productive. A lot of the beneﬁ t of automated tests is

derived during the “debugging phase” of software development, when such tests

can reduce the amount of time spent working with debugging tools. Tackling a

test retroﬁ t on legacy software as your ﬁ rst attempt at automated unit testing

is the last thing you want to try, as it is sure to discourage even the most deter-

mined developers and project managers.

Design for Testability—Upfront

BDUF

design for testability is hard because it is difﬁ cult to know what the tests

will need in the way of control points and observation points on the SUT. We

can easily build software that is difﬁ cult to test. We can also spend a lot of time

designing in testability mechanisms that are either insufﬁ cient or unnecessary.

Either way, we will have spent a lot of effort with nothing to show for it.

Unfortunately, this may result in slower tests when the ﬁ xture is in a database. Never-

theless, it will still be many times faster than if each test had to insert all the records it

needed.

“Big Design Upfront” (also known as “waterfall design”) is the opposite of emergent

design (“just-in-time design”).

How Do We Ensure Testability?

Chapter 6 Test Automation Strategy

Test-Driven Testability

The nice thing about building our software driven by tests is that we don’t have

to think very much about design for testability; we just write the tests and that

forces us to build for testability. The act of writing the test deﬁ nes the control

points and observation points that the SUT needs to provide. Once we have

passed the tests, we know we have a testable design.

Now that I’ve done my bit promoting TDD as a “design for testability” pro-

cess, let’s get on with our discussion of the mechanics of actually making our

software testable.

Control Points and Observation Points

A test interacts with the software

through one or more interfaces or interaction

points. From the test’s point of view, these interfaces can act as either control

points or observation points (Figure 6.6).

Figure 6.6 Control points and observation points. The test interacts with

the SUT through interaction points. Direct interaction points are synchronous

method calls made by the test; indirect interaction points require some form of

Back Door Manipulation. Control points have arrows pointing toward the SUT;

observation points have arrows pointing away from the SUT.

I am deliberately not saying “SUT” here because it interacts with more than just the

SUT.

Fixture

DOC

Setup

Exercise

Verify

Teardown

SUT

Indirect Output

(Observation Point)

Exercise

(with return value)

Get State

Indirect Input

(Control Point)

Indirect Output

(Observation Point)

Direct Inputs

(Control Points)

Direct Outputs

(Observation Points)

Get Something

(with return value)

Something

(no return value)

Initialize

Fixture

DOC

Setup

Exercise

Verify

Teardown

SUT

Indirect Output

(Observation Point)

Exercise

(with return value)

Get State

Indirect Input

(Control Point)

Indirect Output

(Observation Point)

Direct Inputs

(Control Points)

Direct Outputs

(Observation Points)

Get Something

(with return value)

Something

(no return value)

Initialize

A control point is how the test asks the software to do something for it. This

could be for the purpose of putting the software into a speciﬁ c state as part of

setting up or tearing down the test ﬁ xture, or it could be to exercise the SUT.

Some control points are provided strictly for the tests; they should not be used

by the production code because they bypass input validation or short-circuit the

normal life cycle of the SUT or some object on which it depends.

An observation point is how the test ﬁ nds out about the SUT’s behavior dur-

ing the result veriﬁ cation phase of the test. Observation points can be used to

retrieve the post-test state of the SUT or a DOC. They can also be used to

spy on the interactions between the SUT and any components with which it is

expected to interact while it is being exercised. Verifying these indirect outputs

is an example of Back Door Veriﬁ cation (see Back Door Manipulation).

Both control points and observation points can be provided by the SUT as

synchronous method calls; we call this “going in the front door.” Some inter-

action points may be via a “back door” to the SUT; we call this Back Door

Manipulation. In the diagrams that follow, control points are represented by

the arrowheads that point to the SUT, whether from the test or from a DOC.

Observation points are represented by the arrows whose heads point back to

the test itself. These arrows typically start at the SUT or DOC

or start at the

test and interact with either the SUT or DOC before returning to the test.

Interaction Styles and Testability Patterns

When testing a particular piece of software, our tests can take one of two basic

forms.

A round-trip test interacts with the SUT in question only through its public

interface—that is, its “front door” (Figure 6.7). Both the control points and

the observation points in a typical round-trip test are simple method calls. The

nice thing about this approach is that it does not violate encapsulation. The test

needs to know only the public interface of the software; it doesn’t need to know

anything about how it is built.

The main alternative is the layer-crossing test (Figure 6.8), in which we exer-

cise the SUT through the API and keep an eye on what comes out the back door

using some form of Test Double such as a Test Spy (page 538) or Mock Object.

This can be a very powerful testing technique for verifying certain kinds of

mostly architectural requirements. Unfortunately, this approach can also result

in Overspeciﬁ ed Software (see Fragile Test) if it is overused because changes in

how the software implements its responsibilities can cause tests to fail.

An asynchronous observation point.

A synchronous observation point.

How Do We Ensure Testability?