Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

Test Automation Framework

How do we make it easy to write and run tests written by different people?

We use a framework that provides all the mechanisms needed to run the test logic

so the test writer needs to provide only the test-speciﬁ c logic.

Writing and running automated tests involves several steps, but many of these

steps are the same for every test. If every test had to include an implementation

of these steps, writing automated tests would be very tedious, time-consuming,

prone to errors, and expensive.

Using a Test Automation Framework is a way to minimize the effort of writing

Fully Automated Tests (see page 26).

How It Works

We build a framework that implements all the mechanisms required to run suites

of tests and record the results. These mechanisms include the ability to ﬁ nd in-

dividual tests, assemble them into a test suite, execute each test in turn, verify

expected outcomes, collect and report any test failures or errors, and clean up

when failures or errors do occur. The framework provides a way to plug in and

run the test-speciﬁ c behavior that test automaters write.

Test Automation Infrastructure

Fixture

SUT

Test

Script 1

Test

Script 2

Test

Script n

Exercise

Inputs

Test Runner

Setup

Verify

Expected

Outputs

Test Automation Framework

Test Automation Infrastructure

Fixture

SUT

Test

Script 1

Test

Script 2

Test

Script n

Exercise

Inputs

Test Runner

Setup

Verify

Expected

Outputs

Test Automation Framework

Test

Automation

Framework

Chapter 18 Test Strategy Patterns

298

Why We Do This

Building Fully Automated Tests that are repeatable and robust is a much more

complicated process than just writing a test script that invokes the SUT. We need

to handle success cases and error cases, both expected and unexpected. We need

to set up and tear down test ﬁ xtures. We need to specify which test(s) to run. We

also need to report on the results after we have run a suite of tests.

The amount of effort required to build Fully Automated Tests can act as

a serious deterrent to automation of tests. We can reduce the cost of getting

started signiﬁ cantly by providing a framework that implements the most com-

mon functionality—the only entry cost is then incurred while learning to use

the framework. This cost, in turn, can be reduced if the framework implements

a common protocol such as xUnit that makes it easier for us to learn a second

or third framework once we have experience with the ﬁ rst.

Using a framework also helps isolate the implementation of the logic re-

quired to run the tests from the logic of the tests. This approach can help reduce

Test Code Duplication (page 213) and minimize the occurrence of Obscure

Tests (page 186). It also ensures that test written by different test automaters

can be run easily in a single test run with a single report on the test results.

Implementation Notes

Many kinds of Test Automation Frameworks are available, from both com-

mercial vendors and open-source resources. They can be classiﬁ ed into two

main categories: “robot user” test tools and Scripted Tests (page 285). The

latter category can be further subdivided into the xUnit and Data-Driven

Tests (page 288) families of Test Automation Frameworks.

Variation: Robot User Test Frameworks

A large number of third-party test automation tools are designed to test applica-

tions via the user interface. Most of them use the “record and playback” test

metaphor. This metaphor leads to some very seductive marketing materials,

because it makes test automation seem as simple as running some tests manu-

ally while recording the test session. Such a robot user test tool consists of two

major parts: the “test recorder,” which monitors and records the interactions

between the user and the SUT, and the “test runner,” which executes the Recorded

Tests (page 278). Most of these test automation tools are also frameworks that

support a number of “widget recognizer” plug-ins. Most commercial tools come

with a gaggle of built-in widget recognizers.

Test Automation Framework

Test

Automation

Framework

299

Variation: The xUnit Family of Test Automation Frameworks

Most unit-testing tools belong to the xUnit family of testing frameworks

designed for automating Hand-Scripted Tests (see Scripted Test). xUnit has

been ported to (or developed from scratch for) most current programming

languages. The xUnit family of unit-testing frameworks consists of several

major components. The most visible is the Test Runner (page 377), which can

be invoked either from the command line or as a Graphical Test Runner (see

Test Runner). It builds the Testcase Objects (page 382), collects them into Test

Suite Objects (page 387), and invokes each of the Test Methods (page 348).

The other major component of the xUnit frameworks is the library of built-in

Assertion Methods (page 362) that are used within the Test Methods to specify

the expected outcome of each test.

Variation: Data-Driven Test Frameworks

A Data-Driven Test framework provides a way to plug in interpreters that know

how to execute a speciﬁ c kind of test step. This ﬂ exibility, in effect, extends the

format of the input ﬁ le with new “verbs” and objects. Such a framework also

provides a test runner that reads in the ﬁ le, passes control to the plug-ins when

their corresponding data formats are encountered, and keeps track of statistics

for the test run. The most notable member of the Data-Driven Test Frameworks

family is Fit, which enables test automaters to write tests in tabular form and to

“plug in” ﬁ xture classes that know how to interpret speciﬁ c formats of tables.

Example: Test Automation Framework

The Test Automation Framework looks somewhat different for each of the

possible ways to automate tests. To see these variations, refer to Recorded Test,

Scripted Test, and Data-Driven Test for examples of the respective Test Auto-

mation Frameworks.

Further Reading

Some of the more popular examples of Test Automation Frameworks for xUnit

are JUnit (Java), SUnit (Smalltalk), CppUnit (C++), NUnit (all .NET languages),

runit (Ruby), PyUnit (Python), and VbUnit (Visual Basic). A more complete and

up-to-date list can be found at http://xprogramming.com, along with a list of the

available extensions (e.g., HttpUnit, Cactus).

Test

Automation

Framework

Chapter 18 Test Strategy Patterns

300

Other open-source Test Automation Frameworks include Fit, Canoo Web-

Test, and Watir. Commercial Test Automation Frameworks include QTP, BPT,

and eCATT, among many others.

In Test-Driven Development—By Example [TDD-BE], Kent Beck illustrates

TDD by building a Test Automation Framework in Python. In an approach he

likens to “doing brain surgery on yourself,” he uses the emerging Test Automa-

tion Framework to run the tests he writes for each new capability. This applica-

tion is a very good example of both TDD and bootstrapping.

Test Automation Framework

Test

Automation

Framework

301

Minimal Fixture

Which ﬁ xture strategy should we use?

We use the smallest and simplest ﬁ xture possible for each test.

Every test needs some kind of test ﬁ xture. A key part of understanding a test

is understanding the test ﬁ xture and recognizing how it inﬂ uences the expected

outcome of the test. Tests are much easier to understand if the ﬁ xture is small

and simple.

Why We Do This

A Minimal Fixture is important for achieving Tests as Documentation

(see page 23) and for avoiding Slow Tests (page 253). A test that uses a Minimal

Fixture will always be easier to understand than one that uses a ﬁ xture contain-

ing unnecessary or irrelevant information. This is true whether we are using a

Fresh Fixture (page 311) or a Shared Fixture (page 317), although the effort to

build a Minimal Fixture is typically higher with a Shared Fixture because it must

be designed to handle several tests. Deﬁ ning a Minimal Fixture is much easier

for a Fresh Fixture because it need serve only a single test.

Fixture

SUT

Testcase Class

setUp

test_1

test_2

test_n

Fixture

SUT

Testcase Class

setUp

test_1

test_2

test_n

Also known as:

Minimal

Context

Minimal

Fixture

302

Chapter 18 Test Strategy Patterns

Implementation Notes

We design a ﬁ xture that includes only those objects that are absolutely necessary

to express the behavior that the test veriﬁ es. Another way to phrase this is “If

the object is not important to understand the test, it is important not to include

it in the ﬁ xture.”

To build a Minimal Fixture, we ruthlessly remove anything from the ﬁ xture

that does not help the test communicate how the SUT should behave. Two

forms of “minimization” can be considered:

• We can eliminate objects entirely. That is, we don’t even build the

objects as part of the ﬁ xture. If the object isn’t necessary to prove

something about how the SUT behaves, we don’t include it at all.

• We can hide unnecessary attributes of the object when they don’t con-

tribute to the understanding of the expected behavior.

A simple way to ﬁ nd out whether an object is necessary as part of the ﬁ xture

is to remove it. If the test fails as a result, the object was probably necessary

in some way. Of course, it may have been necessary only as an argument to

some method we are not interested in or as an attribute that is never used (even

though the object to which the attribute belongs is required for some reason).

Including these kinds of objects as part of ﬁ xture setup deﬁ nitely contributes to

Obscure Tests (page 186). We can eliminate these unnecessary objects in one of

two ways: (1) by hiding them or (2) by eliminating the need for them by passing

in Dummy Objects (page 728) or using Entity Chain Snipping (see Test Stub

on page 529). If the SUT actually accesses the object as it is executing the logic

under test, however, we may be forced to include the object as part of the test

ﬁ xture.

Having determined that the object is necessary for the execution of the test,

we must now ask whether the object is helpful in understanding the test. If we

were to initialize it “off-stage,” would that make it harder to understand the

test? Would the object lead to an Obscure Test by acting as a Mystery Guest

(see Obscure Test)? If so, we want to keep the object visible. Boundary values

are a good example of a case in which we do want to keep the objects and at-

tributes that take on the boundary values visible.

If we have established that the object or attribute isn’t necessary for

understanding the test, we should make every effort to eliminate it from the

Test Method (page 348), albeit not necessarily from the test ﬁ xture. Creation

Methods (page 415) are a common way of achieving this goal. We can hide

the attributes of objects that don’t affect the outcome of the test but that are

needed for construction of the object by using Creation Methods to ﬁ ll in all

Minimal Fixture

Minimal

Fixture

303

the “don’t care” attributes with meaningful default values. We can also hide

the creation of necessary depended-on objects within the Creation Methods.

A good example of this occurs when we write tests that require badly formed

objects as input (for testing the SUT with invalid inputs). In this case we don’t

want to confuse the issue by showing all valid attributes of the object being

passed to the SUT; there could be many of these extraneous attributes.

Instead, we want to focus on the invalid attribute. To do so, we can use the One

Bad Attribute pattern (see Derived Value on page 718) to build malformed

objects with a minimum of code by calling a Creation Method to construct a

valid object and then replacing a single attribute with the invalid value that

we want to verify the SUT will handle correctly.

Minimal

Fixture

Chapter 18 Test Strategy Patterns

304

Standard Fixture

Which ﬁ xture strategy should we use?

We reuse the design of the text ﬁ xture across the many tests.

To execute an automated test, we require a text ﬁ xture that is well understood

and completely deterministic. Designing a custom test ﬁ xture for each test requires

extra effort. A Standard Fixture offers a way to reuse the same ﬁ xture design in

several tests without necessarily sharing the same ﬁ xture instance.

How It Works

A Standard Fixture is more about attitude than about technology. It requires us

to decide early on in the testing process that we will design a Standard Fixture

that can be used by several or many tests rather than mining a common ﬁ xture

from tests that were designed independently. In a sense, a Standard Fixture is the

result of “Big Design Upfront” of the test ﬁ xture for a whole suite of tests. We

then deﬁ ne our speciﬁ c tests using this common test ﬁ xture design.

The choice of a Standard Fixture is independent of the choice between a

Fresh Fixture (page 311) and a Shared Fixture (page 317). A Shared Fixture

is, by deﬁ nition, a Standard Fixture. The reverse is not true, however, because

a Standard Fixture focuses on reuse of the ﬁ xture’s design—not the time when

the ﬁ xture is built or its visibility. Having chosen to use a Standard Fixture, we

still need to decide whether each test will build its own instance of the Standard

Fixture

Setup

Exercise

Verify

Teardown

SUT

Fixture

Setup

Exercise

Verify

Teardown

SUT

Standard Fixture

Setup Logic

Fixture

Setup

Exercise

Verify

Teardown

SUT

Fixture

Setup

Exercise

Verify

Teardown

SUT

Standard Fixture

Setup Logic

Also known as:

Standard

Context

Standard Fixture

Standard

Fixture

305

Fixture (a Fresh Fixture) or whether we will build it once as a Shared Fixture

and reuse it across many tests.

When to Use It

When I was reviewing an early draft of this book with Series Editor Martin

Fowler, he asked me, “Do people actually do this?” This question exempliﬁ es

the philosophical divide of ﬁ xture design. Coming from an agile background,

Martin lets each test pull a ﬁ xture into existence. If several tests happen to need

the same ﬁ xture, then it makes sense to factor it out into the setUp method and

split the class into one Testcase Class per Fixture (page 631). It doesn’t even occur

to Martin to design a Standard Fixture that all tests can use. So who uses them?

Standard Fixtures are something of a tradition in the testing (quality assess-

ment) community. It is very commonplace to deﬁ ne a large Standard Fixture that

is then used as a test bed for testing activities. This approach makes a lot of sense

in the context of manual execution of many customer tests because it eliminates

the need for each tester to spend a lot of time setting up the test environment for

each customer test and it allows several testers to work in the same test environ-

ment at the same time. Some test automaters also use Standard Fixtures when

deﬁ ning their automated customer tests. This strategy is especially prevalent

when test automaters use a Shared Fixture, for obvious reasons.

In the xUnit community, use of a Standard Fixture simply to avoid designing a

Minimal Fixture (page 302) for each test is considered undesirable and has been

given the name General Fixture (see Obscure Test on page 186). A more accepted

example is the use of Implicit Setup (page 424) in conjunction with Testcase

Class per Fixture because only a few Test Methods (page 348) share the design

of the ﬁ xture and they do so because they need the same design. As we make a

Standard Fixture more reusable across many tests with disparate needs, it tends

to grow larger and more complex. This trend can lead to a Fragile Fixture (see

Fragile Test on page 239) as the needs of new tests introduce changes that break

existing clients of the Standard Fixture. Depending on how we go about building

the Standard Fixture, we may also ﬁ nd ourselves entertaining a Mystery Guest

(see Obscure Test) if the cause–effect relationships between the ﬁ xture and out-

come are not easy to discern either because the ﬁ xture setup is hidden from the

test or because it is not clear which characteristics of the referenced part of the

Standard Fixture serve as pre-conditions for the test.

A Standard Fixture will also take longer to build than a Minimal Fixture

because there is more ﬁ xture to construct. When we are building a Fresh Fixture

for each Testcase Object (page 382), this effort can lead to Slow Tests (page 253),

especially if the ﬁ xture setup involves a database. (See the sidebar “Unit Test Rulz”

Standard

Fixture

306

Chapter 18 Test Strategy Patterns

Unit Test Rulz

Michael Feathers of Object Mentor writes:

I’ve used these rules with a large number of teams. They encour-

age good design and rapid feedback and they seem to help teams

avoid a lot of trouble.

A test is not a unit test if:

• It talks to the database.

• It communicates across the network.

• It touches the ﬁ le system.

• It can’t run correctly at the same time as any of your other unit

tests.

• You have to do special things to your environment (such as

editing conﬁ g ﬁ les) to run it.

Tests that do these things aren’t bad. Often they are worth writ-

ing, and they can be written in a unit test harness. However, it is

important to be able to separate them from true unit tests so that

we can keep a set of tests that we can run fast whenever we make

our changes.

http://www.objectmentor.com

for an opinion about what kinds of behavior are acceptable for a unit test.) For

these reasons, we may be better off using a Minimal Fixture to avoid the extra

ﬁ xture setup overhead associated with creating objects that are only needed in

other tests.

Implementation Notes

As mentioned earlier, we can use a Standard Fixture as either a Fresh Fixture or

a Shared Fixture, and we can set it up using either Implicit Setup or Delegated

Setup (page 411).

When using it as a Fresh Fixture, we can deﬁ ne a Test Utility

Method (page 599) (function or procedure) that builds the Standard Fixture; we

can then call the Test Utility Method from each test that needs this particular design

of ﬁ xture. Alternatively, we can take advantage of xUnit support for Implicit Setup

by putting all of the ﬁ xture construction logic in the setUp method.

Doing it with In-line Setup (page 408) would be silly—we would have to copy the code

to construct the Standard Fixture to every Test Method.

Standard Fixture

Standard

Fixture

307