Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

248

Chapter 16 Behavior Smells

Frequent Debugging

Manual debugging is required to determine the cause of most test failures.

Symptoms

A test run results in a test failure or a test error. The output of the Test Run-

ner (page 377) is insufﬁ cient for us to determine the problem. Thus we have to

use an interactive debugger (or sprinkle print statements throughout the code)

to determine where things are going wrong.

If this case is an exception, we needn’t worry about it. If most test fail-

ures require this kind of debugging, however, we have a case of Frequent

Debugging.

Causes

Frequent Debugging is caused by a lack of Defect Localization (see page 22) in

our suite of automated tests. The failed tests should tell us what went wrong either

through their individual failure messages (see Assertion Message on page 370)

or through the pattern of test failures. If they don’t:

• We may be missing the detailed unit tests that would point out a logic

error inside an individual class.

• We may be missing the component tests for a cluster of classes (i.e., a

component) that would point out an integration error between the indi-

vidual classes. This can happen when we use Mock Objects (page 544)

extensively to replace depended-on objects but the unit tests of the

depended-on objects don’t match the way the Mock Objects are pro-

grammed to behave.

I’ve encountered this problem most frequently when I wrote higher-level (func-

tional or component) tests but failed to write all the unit tests for the individual

methods. (Some people would call this approach storytest-driven development

to distinguish it from unit test-driven development, in which every little bit of

code is pulled into existence by a failing unit test.)

Frequent Debugging can also be caused by Infrequently Run Tests (see Pro-

duction Bugs on page 268). If we run our tests after every little change we

make to the software, we can easily remember what we changed since the last

time we ran the tests. Thus, when a test fails, we don’t have to spend a lot

Also known as:

Manual

Debugging

Frequent

Debugging

249

of time troubleshooting the software to discover where the bug is—we know

where it is because we remember putting it there!

Impact

Manual debugging is a slow, tedious process. It is easy to overlook subtle indi-

cations of a bug and spend many hours tracking down a single logic error. Fre-

quent Debugging reduces productivity and makes development schedules much

less predictable because a single manual debugging session could extend the time

required to develop the software by half a day or more.

Solution Patterns

If we are missing the customer tests for a piece of functionality and manual user

testing has revealed a problem not exposed by any automated tests, we probably

have a case of Untested Requirements (see Production Bugs). We can ask our-

selves, “What kind of automated test would have prevented the manual debug-

ging session?” Better yet, once we have identiﬁ ed the problem, we can write a

test that exposes it. Then we can use the failing test to do test-driven bug ﬁ xing.

If we suspect this to be a widespread problem, we can create a development task

to identify and write any additional tests that would be required to ﬁ ll the gap

we just exposed.

Doing true test-driven development is the best way to avoid the circumstances

that lead to Frequent Debugging. We should start as close as possible to the

skin of the application and do storytest-driven development—that is, we should

write unit tests for individual classes as well as component tests for the collec-

tions of related classes to ensure we have good Defect Localization.

Frequent Debugging

Frequent

Debugging

250

Chapter 16 Behavior Smells

Manual Intervention

A test requires a person to perform some manual action

each time it is run.

Symptoms

The person running the test must do something manually either before the test

is run or partway through the test run; otherwise, the test fails. The Test Runner

may need to verify the results of the test manually.

Impact

Automated tests are all about getting early feedback on problems introduced

into the software. If the cost of getting that feedback is too high—that is, if it

takes the form of Manual Intervention—we likely won’t run the tests very often

and we won’t get the feedback very often. If we don’t get that feedback very

often, we’ll probably introduce lots of problems between test runs, which will

ultimately lead to Frequent Debugging (page 248) and High Test Maintenance

Cost (page 265).

Manual Intervention also makes it impractical to have a fully automated

Integration Build [SCM] and regression test process.

Causes

The causes of Manual Intervention are as varied as the kinds of things our soft-

ware does or encounters. The following are some general categories of the kinds

of issues that require Manual Intervention. This list is by no means exhaustive,

though.

Cause: Manual Fixture Setup

Symptoms

A person has to set up the test environment manually before the automated tests

can be run. This activity may take the form of conﬁ guring servers, starting server

processes, or running scripts to set up a Prebuilt Fixture (page 429).

Root Cause

This problem is typically caused by a lack of attention to automating the ﬁ xture

setup phase of the test. It may also be caused by excessive coupling between

Manual

Intervention

251

components in the SUT that prevents us from testing a majority of the code in

the system inside the development environment.

Possible Solution

We need to make sure that we are writing Fully Automated Tests. This may

require opening up test-speciﬁ c APIs to allow tests to set up the ﬁ xture. Where

the issue is related to an inability to run the software in the development envi-

ronment, we may need to refactor the software to decouple the SUT from the

steps that would otherwise need to be done manually.

Cause: Manual Result Veriﬁ cation

Symptoms

We can run the tests but they almost always pass—even when we know that the

SUT is not returning the correct results.

Root Cause

If the tests we write are not Self-Checking Tests (see page 26), we can be given a

false sense of security because tests will fail only if an error/exception is thrown.

Possible Solution

We can ensure that our tests are all self-checking by including result veriﬁ ca-

tion logic such as calls to Assertion Methods (page 362) within the Test Meth-

ods (page 348).

Cause: Manual Event Injection

Symptoms

A person must intervene during test execution to perform some manual action

before the test can proceed.

Root Cause

Many events in a SUT are hard to generate under program control. Examples

include unplugging network cables, bringing down database connections, and

clicking buttons on a user interface.

Impact

If a person needs to do something manually, it both increases the effort to run

the test and ensures that the test cannot be run unattended. This torpedoes any

attempt to do a fully automated build-and-test cycle.

Manual Intervention

Manual

Intervention

252

Chapter 16 Behavior Smells

Possible Solution

The best solution is to ﬁ nd ways to test the software that do not require a real

person to do the manual actions. If the events are reported to the SUT through

asynchronous events, we can have the Test Method invoke the SUT directly,

passing it a simulated event object. If the SUT experiences the situation as a syn-

chronous response from some other part of the system, we can get control of the

indirect inputs by replacing some part of the SUT with a Test Stub (page 529)

that simulates the circumstances to which we want to expose the SUT.

Further Reading

Refer to Chapter 11, Using Test Doubles, for a much more detailed description

of how to get control of the indirect inputs of the SUT.

Manual

Intervention

253

Slow Tests

The tests take too long to run.

Symptoms

The tests take long enough to run that developers don’t run them every time they make

a change to the SUT. Instead, the developers wait until the next coffee break or another

interruption before running them. Or, whenever they run the tests, they walk around

and chat with other team members (or play Doom or surf the Internet or . . .).

Impact

Slow Tests obviously have a direct cost: They reduce the productivity of the

person running the test. When we are test driving the code, we’ll waste precious

seconds every time we run our tests; when it is time to run all the tests before we

commit our changes, we’ll have an even more signiﬁ cant wait time.

Slow Tests also have many indirect costs:

• The bottleneck created by holding the “integration token” longer because

we need to wait for the tests to run after merging all our changes.

• The time during which other people are distracted by the person wait-

ing for his or her test run to ﬁ nish.

• The time spent in debuggers ﬁ nding a problem that was inserted

sometime after the last time we ran the test. The longer it has been

since the test was run, the less likely we are to remember exactly what

we did to break the test. This cost is a result of the breakdown of the

rapid feedback that automated unit tests provide.

A common reaction to Slow Tests is to immediately go for a Shared Fix-

ture (page 317). Unfortunately, this approach almost always results in other

problems, including Erratic Tests (page 228). A better solution is to use a Fake

Object (page 551) to replace slow components (such as the database) with faster

ones. However, if all else fails and we must use some kind of Shared Fixture, we

should make it immutable if at all possible.

Troubleshooting Advice

Slow Tests can be caused either by the way the SUT is built and tested or by

the way the tests are designed. Sometimes the problem is obvious—we can just

Slow Tests

254

Chapter 16 Behavior Smells

watch the green bar grow as we run the tests. There may be notable pauses in the

execution; we may see explicit delays coded in a Test Method (page 348). If the

cause is not obvious, however, we can run different subsets (or subsuites) of tests

to see which ones run quickly and which ones take a long time to run.

A proﬁ ling tool can come in handy to see where we are spending the extra

time in test execution. Of course, xUnit gives us a simple means to build our

own mini-proﬁ ler: We can edit the setUp and tearDown methods of our Testcase

Superclass (page 638). We then write out the start/end times or test duration

into a log ﬁ le, along with the name of the Testcase Class (page 373) and Test

Method. Finally, we import this ﬁ le into a spreadsheet, sort by duration, and

voila—we have found the culprits. The tests with the longest execution times

are the ones on which it will be most worthwhile to focus our efforts.

Causes

The speciﬁ c cause of the Slow Tests could lie either in how we built the SUT or

in how we coded the tests themselves. Sometimes, the way the SUT was built

forces us to write our tests in a way that makes them slow. This is particularly a

problem with legacy code or code that was built with a “test last” perspective.

Cause: Slow Component Usage

A component of the SUT has high latency.

Root Cause

The most common cause of Slow Tests is interacting with a database in many of

the tests. Tests that have to write to a database to set up the ﬁ xture and read a

database to verify the outcome (a form of Back Door Manipulation; see page 327)

take about 50 times longer to run than the same tests that run against in-memory

data structures. This is an example of the more general problem of using slow

components.

Possible Solution

We can make our tests run much faster by replacing the slow components with

a Test Double (page 522) that provides near-instantaneous responses. When the

slow component is the database, the use of a Fake Database (see Fake Object)

can make the tests run on average 50 times faster! See the sidebar “Faster Tests

Without Shared Fixtures” on page 319 for other ways to skin this cat.

Slow Tests

255

Cause: General Fixture

Symptoms

Tests are consistently slow because each test builds the same over-engineered

ﬁ xture.

Root Cause

Each test constructs a large General Fixture each time a Fresh Fixture (page 311)

is built. Because a General Fixture contains many more objects than a Mini-

mal Fixture (page 302), it naturally takes longer to construct. Fresh Fixture

involves setting up a brand-new instance of the ﬁ xture for each Testcase Object

(page 382), so multiply “longer” by the number of tests to get an idea of the

magnitude of the slowdown!

Possible Solution

Our ﬁ rst inclination is often to implement the General Fixture as a Shared Fix-

ture to avoid rebuilding it for each test. Unless we can make this Shared Fixture

immutable, however, this approach is likely to lead to Erratic Tests and should

be avoided. A better solution is to reduce the amount of ﬁ xture setup performed

by each test.

Cause: Asynchronous Test

Symptoms

A few tests take inordinately long to run; those tests contain explicit delays.

Root Cause

Delays included within a Test Method slow down test execution considerably.

This slow execution may be necessary when the software we are testing spawns

threads or processes (Asynchronous Code; see Hard-to-Test Code on page 209)

and the test needs to wait for them to launch, run, and verify whatever side ef-

fects they were expected to have. Because of the variability in how long it takes

for these threads or processes to be started, the test usually needs to include

a long delay “just in case”—that is, to ensure it passes consistently. Here’s an

example of a test with delays:

Slow Tests

256

Chapter 16 Behavior Smells

public class RequestHandlerThreadTest extends TestCase {

private static ﬁnal int TWO_SECONDS = 3000;

public void testWasInitialized_Async() = throws InterruptedException {

// Setup

RequestHandlerThread sut = new RequestHandlerThread();

// Exercise

sut.start();

// Verify

Thread.sleep(TWO_SECONDS);

assertTrue(sut.initializedSuccessfully());

}

public void testHandleOneRequest_Async()

throws InterruptedException {

// Setup

RequestHandlerThread sut = new RequestHandlerThread();

sut.start();

// Exercise

enqueRequest(makeSimpleRequest());

// Verify

Thread.sleep(TWO_SECONDS);

assertEquals(1, sut.getNumberOfRequestsCompleted());

assertResponseEquals(makeSimpleResponse(), getResponse());

}

Impact

A two-second delay might not seem like a big deal. But consider what happens

when we have a dozen such tests: It would take almost half a minute to run these

tests. In contrast, we can run several hundred normal tests each second.

Possible Solution

The best way to address this problem is to avoid asynchronicity in tests by test-

ing the logic synchronously. This may require us to do an Extract Testable Com-

ponent (page 767) refactoring to implement a Humble Executable (see Humble

Object on page 695).

Cause: Too Many Tests

Symptoms

There are so many tests that they are bound to take a long time to run regardless

of how fast they execute.

Slow Tests

257

Root Cause

The obvious cause of this problem is having so many tests. Perhaps we have such

a large system that the large number of tests really is necessary, or perhaps we

have too much overlap between tests.

The less obvious cause is that we are running too many of the tests too fre-

quently!

Possible Solution

We don’t have to run all the tests all the time! The key is to ensure that all tests

are run regularly. If the entire suite is taking too long to run, consider creating

a Subset Suite (see Named Test Suite on page 592) with a suitable cross section

of tests; run this subsuite before every commit operation. The rest of the tests

can be run regularly, albeit less often, by scheduling them to run overnight or at

some other convenient time. Some people call this technique a “build pipeline.”

For more on this and other ideas, see the sidebar “Faster Tests Without Shared

Fixtures” on page 319.

If the system is large in size, it is a good idea to break it into a number

of fairly independent subsystems or components. This allows teams work-

ing on each component to work independently and to run only those tests

specific to their own component. Some of those tests should act as proxies

for how the other components would use the component; they must be kept

up-to-date if the interface contract changes. Hmmm, Tests as Documenta-

tion (see page 23); I like it! Some end-to-end tests that exercise all the com-

ponents together (likely a form of storytests) would be essential, but they

don’t need to be included in the pre-commit suite.

Slow Tests