Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

When to Use It

Regardless of why we use them, Shared Fixtures come with some baggage that

we should understand before we head down this path. The major issue with a

Shared Fixture is that it can lead to interactions between tests, possibly resulting

in Erratic Tests (page 228) if some tests depend on the outcomes of other tests.

Another potential problem is that a ﬁ xture designed to serve many tests is bound

to be much more complicated than the Minimal Fixture (page 302) needed for a

single test. This greater complexity will typically take more effort to design and

can lead to a Fragile Fixture (see Fragile Test on page 239) later on down the

road when we need to modify the ﬁ xture.

A Shared Fixture will often result in an Obscure Test (page 186) because

the ﬁ xture is not constructed inside the test. This potential disadvantage can be

mitigated by using Finder Methods (see Test Utility Method on page 599) with

Intent-Revealing Names [SBPP] to access the relevant parts of the ﬁ xture.

There are some valid reasons for using a Shared Fixture and some misguided

ones. Many of the variations have been devised primarily to mitigate the negative

consequences of using a Shared Fixture. So, what are good reasons for using a

Shared Fixture?

Variation: Slow Tests

We can use a Shared Fixture when we cannot afford to build a new Fresh Fixture

for each test. Typically, this scenario will occur when it takes too much processing

to build a new ﬁ xture for each test, which often leads to Slow Tests (page 253).

It most commonly occurs when we are testing with real test databases due to

the high cost of creating each of the records. This growth in overhead tends to

be exacerbated when we use the API of the SUT to create the reference data,

because the SUT often does a lot of input validation, which may involve reading

some of the just-written records.

A better solution is to make the tests run faster by not interacting with the

database at all. For a more complete list of options, see the solutions to Slow

Tests and the sidebar “Faster Tests Without Shared Fixtures” (page 319).

Shared

Fixture

318

Chapter 18 Test Strategy Patterns

Faster Tests Without Shared Fixtures

The ﬁ rst reaction to Slow Tests (page 253) is often to switch to a Shared

Fixture (page 317) approach. Several other solutions are available, how-

ever. This sidebar describes some experiences on several projects.

Fake Database

On one of our early XP projects, we wrote a lot of tests that accessed

the database. At ﬁ rst we used a Shared Fixture. When we encountered

Interacting Tests (see Erratic Test on page 228) and later Test Run Wars

(see Erratic Test), however, we changed to a Fresh Fixture (page 311)

approach. Because these tests needed a fair bit of reference data, they

were taking a long time to run. On average, for every read or write the

SUT did to or from the database, each test did several more. It was tak-

ing 15 minutes to run the full test suite of several hundred tests, which

greatly impeded our ability to integrate our work quickly and often.

At the time, we were using a data access layer to keep the SQL out of

our code. We soon discovered that it allowed us to replace the real data-

base with a functionally equivalent Fake Database (see Fake Object on

page 551). We started out by using simple

HashTables to store the objects

against a key. This approach allowed us to run many of our simpler tests

“in memory” rather than against the database. And that bought us a sig-

niﬁ cant drop in test execution time.

Our persistence framework supported an object query interface. We were

able to build an interpreter of the object queries that ran against our

HashTable database implementation and that allowed the majority of our

tests to work entirely in memory. On average, our tests ran about 50 times

faster in memory than with the database. For example, a test suite that took

10 minutes to run with the database took 10 seconds to run in memory.

This approach was so successful that we have reused the same testing

infrastructure on many of our subsequent projects. Using the faked-out

persistence framework also means we don’t have to bother with building

a “real database” until our object models stabilize, which can be several

months into the project.

Incremental Speedups

Ted O’Grady and Joseph King are agile team leads on a large (50-plus

developers, subject matter experts, and testers) eXtreme Programming

project. Like many project teams building database-centric applications,

Shared Fixture

Shared

Fixture

319

Continued...

they suffered from Slow Tests. But they found a way around this problem:

As of late 2005, their check-in test suite ran in less than 8 minutes com-

pared to 8 hours for a full test run against the database. That is a pretty

impressive speed difference. Here is their story:

Currently we have about 6,700 tests that we run on a regular

basis. We’ve actually tried a few things to speed up the tests and

they’ve evolved over time.

In January 2004, we were running our tests directly against a

database via Toplink.

In June 2004, we modiﬁ ed the application so we could run tests

against an in-memory, in-process Java database (HSQL). This cut

the time to run in half.

In August 2004, we created a test-only framework that allowed

Toplink to work without a database at all. That cut the time to

run all the tests by a factor of 10.

In July 2005, we built a shared “check-in” test execution server

that allowed us to run tests remotely. This didn’t save any time at

ﬁ rst but it has proven to be quite useful nonetheless.

In July 2005, we also started using a clustering framework that al-

lowed us to run tests distributed across a network. This cut the

time to run the tests in half.

In August 2005, we removed the GUI and Master Data (reference data

crud) tests from the “check-in suite” and ran them only from Cruise

Control. This cut the time to run by approximately 15% to 20%.

Since May 2004, we have also had Cruise Control run all the tests

against the database at regular intervals. The time it takes Cruise

Control to complete [the build and run the tests] has grown with

the number of tests from an hour to nearly 8 hours now.

When a threshold has been met that prevents the developers from

(a) running [the tests] frequently when developing and (b) creat-

ing long check-in queues as people wait for the token to check in,

we have adapted by experimenting with new techniques. As a rule

we try to keep the running of the tests under 5 minutes, with any-

thing over 8 minutes being a trigger to try something new.

We have resisted thus far the temptation to run only a subset of

the tests and instead focused on ways to speed up running all the

tests—although as you can see, we have begun removing the tests

Shared

Fixture

320

Chapter 18 Test Strategy Patterns

developers must run continuously (e.g., Master Data and GUI

test suites are not required to check in, as they are run by Cruise

Control and are areas that change infrequently).

Two of the most interesting solutions recently (aside from the in-

memory framework) are the test server and the clustering frame-

work.

The test server (named the “check-in” box here) is actually quite

useful and has proven to be reliable and robust. We bought an

Opteron box that is roughly twice as fast as the development

boxes (really, the fastest box we could ﬁ nd). The server has an

account set up for each development machine in the pit. Using

the UNIX tool rsynch, the Eclipse workspace is synchronized

with the user’s corresponding server account ﬁ le system. A series

of shell scripts then recreates the database on the server for the

remote account and runs all the development tests. When the tests

have completed, a list of times to run each test is dumped to the

console, along with a MyTestSuite.java class containing all the

test failures, which the developer can use to run locally to ﬁ x any

tests that have broken. The biggest advantage the remote server

has provided is that it makes running a large number of tests feel

fast again, because the developer can continue working while he

or she waits for the results of the test server to come back.

The clustering framework (based on Condor) was quite fast but had

the defect that it had to ship the entire workspace (11MB) to all the

nodes on the network (×20), which had a signiﬁ cant cost, especially

when a dozen pairs are using it. In comparison, the test server uses

rsynch, which copies only the ﬁ les that are new or different in the

developer’s workspace. The clustering framework also proved to be

less reliable than the server solution, frequently not returning any

status of the test run. There were also some tests that would not run

reliably on the framework. Since it gave us roughly the same perfor-

mance as the “check-in” test server, we have put this solution on the

back burner.

Further Reading

A more detailed description of the ﬁ rst experience can be found at http://

FasterTestsPaper.gerardmeszaros.com.

Shared Fixture

Shared

Fixture

321

Variation: Incremental Tests

We may also use Shared Fixtures when we have a long, complex sequence of

actions, each of which depends on the previous actions. In customer tests, this

may show up as a work ﬂ ow; in unit tests, it may be a sequence of method calls

on the same object. This case might be tested using a single Eager Test (see As-

sertion Roulette on page 224). The alternative is to put each distinct action into

a separate Test Method (page 348) that builds upon the actions of a previous test

operating on a Shared Fixture. This approach, which is an example of Chained

Tests (page 454), is how testers in the “testing” (i.e., QA) community often

operate: They set up a ﬁ xture and then run a sequence of tests, each of which

builds upon the ﬁ xture. The testers do have one signiﬁ cant advantage over our

Fully Automated Tests (see page 26): When a test partway through the chain

fails, they are available to make decisions about how to recover or whether it is

worth proceeding at all. In contrast, our automated tests just keep running, and

many of them will generate test failures or errors because they did not ﬁ nd the

ﬁ xture as expected and, therefore, the SUT behaved (probably correctly) differ-

ently. The resulting test results can obscure the real cause of the failure in a sea

of red. With some experience it is often possible to recognize the failure pattern

and deduce the root cause.

This troubleshooting can be made simpler by starting each Test Method with

one or more Guard Assertions (page 490) that document the assumptions the

Test Method makes about the state of the ﬁ xture. When these assertions fail,

they tell us to look elsewhere—either at tests that failed earlier in the test suite

or at the order in which the tests were run.

Implementation Notes

A key implementation question with Shared Fixtures is, How do tests know about

the objects in the Shared Fixture so they can (re)use them? Because the point of

a Shared Fixture is to save execution time and effort by having multiple tests use

the same instance of the test ﬁ xture, we’ll need to keep a reference to the ﬁ xture

we create. That way, we can ﬁ nd the ﬁ xture if it already exists and we can inform

other tests that it now exists once we have constructed it. We have more choices

available to us with Per-Run Fixtures because we can “remember” the ﬁ xture we

set up in code more easily than a Prebuilt Fixture (page 429) set up by a different

program. Although we could just hard-code the identiﬁ ers (e.g., database keys) of

the ﬁ xture objects into all our tests, that technique would result in a Fragile Fix-

ture. To avoid this problem, we need to keep a reference to the ﬁ xture when we

create it and we need to make it possible for all tests to access that reference.

It may not be as simple as looking at the ﬁ rst test that failed.

Shared

Fixture

Chapter 18 Test Strategy Patterns

322

Variation: Per-Run Fixture

The simplest form of Shared Fixture is the Per-Run Fixture, in which we set up

the ﬁ xture at the beginning of a test run and allow it to be shared by the tests

within the run. Ideally, the ﬁ xture won’t outlive the test run and we don’t have

to worry about interactions between test runs such as Unrepeatable Tests (a

cause of Erratic Tests). If the ﬁ xture is persistent, such as when it is stored in a

database, we may need to do explicit ﬁ xture teardown.

If a Per-Run Fixture is shared only within a single Testcase Class (page 373),

the simplest solution is to use a class variable for each ﬁ xture object we need to

hold a reference to and then use either Lazy Setup (page 435) or Suite Fixture

Setup (page 441) to initialize the objects just before we run the ﬁ rst test in the

suite. If we want to share the test ﬁ xture between many Testcase Classes, we’ll

need to use a Setup Decorator (page 447) to hold the setUp and tearDown methods

and a Test Fixture Registry (see Test Helper on page 643) (which could just be

the test database) to access the ﬁ xture.

Variation: Immutable Shared Fixture

The problem with Shared Fixtures is that they lead to Erratic Tests if tests modify

the Shared Fixture (page 317). Shared Fixtures violate the Independent Test prin-

ciple (see page 42). We can avoid this problem by making the Shared Fixture

immutable; that is, we partition the ﬁ xture needed by tests into two logical parts.

The ﬁ rst part is the stuff every test needs to have present but is never modiﬁ ed by

any tests—that is, the Immutable Shared Fixture. The second part is the objects

that any test needs to modify or delete; these objects should be built by each test

as Fresh Fixtures.

The most difﬁ cult part of applying an Immutable Shared Fixture is deciding

what constitutes a change to an object. The key guideline is this: If any test per-

ceives something done by another test as a change to an object in the Immutable

Shared Fixture, then that change shouldn’t be allowed in any test with which it

shares the ﬁ xture. Most commonly, the Immutable Shared Fixture consists of

reference data that is needed by the actual per-test ﬁ xtures. The per-test ﬁ xtures

can then be built as Fresh Fixtures on top of the Immutable Shared Fixture.

Motivating Example

The following example shows a Testcase Class setting up the test ﬁ xture via

Implicit Setup (page 424). Each Test Method uses an instance variable to access

the contents of the ﬁ xture.

Shared Fixture

Shared

Fixture

323

public void testGetFlightsByFromAirport_OneOutboundFlight()

throws Exception {

setupStandardAirportsAndFlights();

FlightDto outboundFlight = ﬁndOneOutboundFlight();

// Exercise System

List ﬂightsAtOrigin = facade.getFlightsByOriginAirport(

outboundFlight.getOriginAirportId());

// Verify Outcome

assertOnly1FlightInDtoList( "Flights at origin",

outboundFlight,

ﬂightsAtOrigin);

}

public void testGetFlightsByFromAirport_TwoOutboundFlights()

throws Exception {

setupStandardAirportsAndFlights();

FlightDto[] outboundFlights =

ﬁndTwoOutboundFlightsFromOneAirport();

// Exercise System

List ﬂightsAtOrigin = facade.getFlightsByOriginAirport(

outboundFlights[0].getOriginAirportId());

// Verify Outcome

assertExactly2FlightsInDtoList( "Flights at origin",

outboundFlights,

ﬂightsAtOrigin);

}

Note that the setUp method is run once for each Test Method. If the ﬁ xture setup

is fairly complex and involves accessing a database, this approach could result

in Slow Tests.

Refactoring Notes

To convert a Testcase Class from a Standard Fixture to a Shared Fixture, we

simply convert the instance variables into class variables to make the ﬁ xture

outlast the creating Testcase Object. We then need to initialize the class vari-

ables just once to avoid recreating them for each Test Method; Lazy Setup is an

easy way to accomplish this task. Of course, other ways to set up the Shared

Fixture are also possible, such as Setup Decorator or Suite Fixture Setup.

Example: Shared Fixture

This example shows the ﬁ xture converted to a Shared Fixture set up using Lazy

Setup.

protected void setUp() throws Exception {

if (sharedFixtureInitialized) {

return;

Shared

Fixture

324

Chapter 18 Test Strategy Patterns

}

facade = new FlightMgmtFacadeImpl();

setupStandardAirportsAndFlights();

sharedFixtureInitialized = true;

}

protected void tearDown() throws Exception {

// We cannot delete any objects because we don't know

// whether this is the last test

}

The Lazy Initialization [SBPP] logic in the setUp method ensures that the Shared

Fixture is created whenever the class variable is uninitialized. The Test Methods

have also been modiﬁ ed to use a Finder Method to access the contents of the

ﬁ xture:

public void testGetFlightsByFromAirport_OneOutboundFlight()

throws Exception {

FlightDto outboundFlight = ﬁndOneOutboundFlight();

// Exercise System

List ﬂightsAtOrigin = facade.getFlightsByOriginAirport(

outboundFlight.getOriginAirportId());

// Verify Outcome

assertOnly1FlightInDtoList( "Flights at origin",

outboundFlight,

ﬂightsAtOrigin);

}

public void testGetFlightsByFromAirport_TwoOutboundFlights()

throws Exception {

FlightDto[] outboundFlights =

ﬁndTwoOutboundFlightsFromOneAirport();

// Exercise System

List ﬂightsAtOrigin = facade.getFlightsByOriginAirport(

outboundFlights[0].getOriginAirportId());

// Verify Outcome

assertExactly2FlightsInDtoList( "Flights at origin",

outboundFlights,

ﬂightsAtOrigin);

}

The details of how the Test Utility Methods such as setupStandardAirportsAndFlights

are implemented are not shown here, because they are not important for under-

standing this example. It should be enough to understand that these methods

create the airports and ﬂ ights and store references to them in static variables

so that all Test Methods can access the same ﬁ xture either directly or via Test

Utility Methods.

Shared Fixture

Shared

Fixture

325

Example: Immutable Shared Fixture

Here’s an example of Shared Fixture “pollution”:

public void testCancel_proposed_p()throws Exception {

// shared ﬁxture

BigDecimal proposedFlightId = ﬁndProposedFlight();

// exercise SUT

facade.cancelFlight(proposedFlightId);

// verify outcome

try{

assertEquals(FlightState.CANCELLED,

facade.ﬁndFlightById(proposedFlightId));

} ﬁnally {

// teardown

// try to undo the damage; hope this works!

facade.overrideStatus( proposedFlightId,

FlightState.PROPOSED);

}

We can avoid this problem by making the Shared Fixture immutable; that is, we

partition the ﬁ xture needed by tests into two logical parts. The ﬁ rst part is the

stuff every test needs to have present but is never modiﬁ ed by any tests—that is,

the Immutable Shared Fixture. The second part is the objects that any test needs

to modify or delete; these objects should be built by each test as Fresh Fixtures.

Here’s the same test modiﬁ ed to use an Immutable Shared Fixture. We simply

created our own mutableFlight within the test.

public void testCancel_proposed() throws Exception {

// ﬁxture setup

BigDecimal mutableFlightId =

createFlightBetweenInsigiﬁcantAirports();

// exercise SUT

facade.cancelFlight(mutableFlightId);

// verify outcome

assertEquals( FlightState.CANCELLED,

facade.ﬁndFlightById(mutableFlightId));

// teardown

// None required because we let the SUT create

// new IDs for each ﬂight. We might need to clean out

// the database eventually.

}

Note that we don’t need any ﬁ xture teardown logic in this version of the test because

the SUT uses a Distinct Generated Value (see Generated Value on page 723)—that

is, we do not supply a ﬂ ight number. We also use the predeﬁ ned dummyAirport1 and

dummyAirport2 to avoid changing the number of ﬂ ights for airports used by other

tests. Therefore, the mutable ﬂ ights can accumulate in the database trouble-free.

Shared

Fixture

326

Chapter 18 Test Strategy Patterns

Back Door Manipulation

How can we verify logic independently when we cannot use

a round-trip test?

We set up the test ﬁ xture or verify the outcome by going through a back door

(such as direct database access).

Every test requires a starting point (the test ﬁ xture) and an expected ﬁ nishing

point (the expected results). The “normal” approach is to set up the ﬁ xture and

verify the outcome by using the API of the SUT itself. In some circumstances this

is either not possible or not desirable.

In some situations we can use Back Door Manipulation to set up the ﬁ xture

and/or verify the SUT’s state.

How It Works

The state of the SUT comes in many ﬂ avors. It can be stored in memory, on disk

as ﬁ les, in a database, or in other applications with which the SUT interacts.

Whatever form it takes, the pre-conditions of a test typically require that the

state of the SUT is not just known but is a speciﬁ c state. Likewise, at the end of

the test we often want to do State Veriﬁ cation (page 462) of the SUT’s state.

If we have access to the state of the SUT from outside the SUT, the test can

set up the pre-test state of the SUT by bypassing the normal API of the SUT

and interacting directly with whatever is holding that state via a “back door.”

When exercising of the SUT has been completed, the test can similarly access

Data

Fixture

Setup

Exercise

Verify

Teardown

SUT

Data

Fixture

Setup

Exercise

Verify

Teardown

SUT

Back Door Manipulation

Back Door

Manipulation

327

Also known as:

Layer-Crossing

Test