Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

268

Chapter 17 Project Smells

Production Bugs

We ﬁ nd too many bugs during formal tests or in production.

Symptoms

We have put a lot of effort into writing automated tests, yet the number of bugs

showing up in formal (i.e., system) testing or production remains too high.

Impact

It takes longer to troubleshoot and ﬁ x bugs found in formal testing than those

found in development, and even longer to troubleshoot and ﬁ x bugs found in

production. We may be forced to delay shipping the product or putting the

application into production to allow time for the bug ﬁ xes and retesting. This

time and effort translate directly into monetary costs and consume resources

that might otherwise be used to add more functionality to the product or to

build other products. The delay may also damage the organization’s credibility

in the eyes of its customers. Poor quality has an indirect cost as well, in that it

lowers the value of the product or service we are supplying.

Causes

Bugs may slip through to production for several reasons, including Infrequently

Run Tests or Untested Code. The latter problem may result from Missing Unit

Tests or Lost Tests.

By specifying that “enough tests” be run, we mean the test coverage should

be adequate, rather than that some speciﬁ c number of tests must be carried out.

Changes to Untested Code are more likely to result in Production Bugs because

there are no automated tests to tell the developers when they have introduced

problems. Untested Requirements aren’t being veriﬁ ed every time the tests are

run, so we don’t know for sure what is working. Both of these problems are

related to Developers Not Writing Tests (page 263).

Cause: Infrequently Run Tests

Symptoms

We hear that our developers aren’t running the tests very often. When we ask some

questions, we discover that running the tests takes too long (Slow Tests; see page 253)

or produces too many extraneous failures (Buggy Tests; see page 260).

Production

Bugs

269

We see test failures in the daily Integration Build [SCM]. When we dig deeper,

we ﬁ nd that developers often commit their code without running the tests on

their own machines.

Root Cause

Once they’ve seen the beneﬁ ts of working with the safety net of automated tests,

most developers will continue using these tests unless something gets in the way.

The most common impediments are Slow Tests that slow down the pre-integration

regression testing or Unrepeatable Tests (see Erratic Test on page 228) that force

developers to restart their test environment or do Manual Intervention (page 250)

before running the tests.

Possible Solution

If the root cause is Unrepeatable Tests, we can try switching to a Fresh Fix-

ture (page 311) strategy to make the tests more deterministic. If the cause is Slow

Tests, we must put more effort into speeding up the test run.

Cause: Lost Test

Symptoms

The number of tests being executed in a test suite has declined (or has not

increased as much as expected). We may notice this directly if we are paying

attention to test counts. Alternatively, we may ﬁ nd a bug that should have been

caused by a test that we know exists but, upon poking around, we discover that

the test has been disabled.

Root Cause

Lost Tests can be caused by either a Test Method (page 348) or a Testcase

Class (page 373) that has been disabled or has never been added to the AllTests

Suite (see Named Test Suite on page 592).

Tests can be accidentally left out (i.e., never run) of test suite in the following

circumstances:

• We forget to add the [test] attribute to the Test Method, or we acci-

dentally use a method name that doesn’t match the naming convention

used by the Test Discovery (page 393) mechanism.

• We forget to add a call to suite.addTest to add the Test Method to the Test

Suite Object (page 387) when we are automating tests in a Test Automation

Framework (page 298) that supports only Test Enumeration (page 399).

Production Bugs

Production

Bugs

270

Chapter 17 Project Smells

• We forget to add a call to the Test Method explicitly in the Test Suite

Procedure (see Test Suite Object) in procedural-language variations of

xUnit.

• We forget to add the test suite to the Suite of Suites (see Test Suite Object)

or forget to add the [Test Fixture] attribute to the Testcase Class.

Tests that ran in the past may have been disabled in any of the following ways:

• We renamed the Test Method to not match the pattern that causes Test

Discovery to include the test in the test suite (e.g., the method name

starts with “test . . .”).

• We added an [Ignore] attribute in variants of xUnit that use method

attributes to indicate Test Methods.

• We commented out (or deleted) the code that adds the test (or suite) to

the suite explicitly.

Typically, a Lost Test occurs when a test is failing and someone disables it to

avoid having to wade through the failing tests when running other tests. It may

also occur accidentally, of course.

Possible Solution

There are a number of ways to avoid introducing Lost Tests.

We can use a Single Test Suite (see Named Test Suite) to run a single Test

Method instead of disabling the failing or slow test. We can use the Test Tree

Explorer (see Test Runner on page 377) to drill down and run a single test

from within a test suite. Both of these techniques are made difﬁ cult by Chained

Tests (page 454)—a deliberate form of Interacting Tests (see Erratic Test)—so

this is just one more reason to avoid them.

If our variant of xUnit supports it, we can use the provided mechanism to

ignore

a test. It will typically remind us of the number of tests not being run

so we don’t forget to re-enable them. We can also conﬁ gure our continuous

integration tool to fail the build if the number of tests “ignored” exceeds a

certain threshold.

We can compare the number of tests we have after check-in with the number

of tests that existed in the code branch immediately before we started integra-

tion. We simply verify that this count has increased by the number of tests we

have added.

For example, NUnit lets us put the attribute [Ignore] on a Test Method to keep it from

being run.

Production

Bugs

271

We can implement or take advantage of Test Discovery if our programming

language supports reﬂ ection.

We can use a different strategy for ﬁ nding the tests to run in the Integration

Build. Some build tools (such as Ant) let us ﬁ nd all ﬁ les that match a name pat-

tern (e.g., those ending in “Test”). We won’t lose entire test suites if we use this

capability to pick up all the tests.

Cause: Missing Unit Test

Symptoms

All the unit tests pass but a customer test continues to fail. At some point, the

customer test passed—but no unit tests were written to verify the behavior of the

individual classes. Then, a subsequent code change modiﬁ ed the behavior of one of

the classes, which broke its functionality.

Root Cause

Missing Unit Tests often happen when a team focuses on writing the customer

tests but fails to do test-driven development using unit tests. The team members

may have built enough functionality to pass the customer tests, but a subsequent

refactoring broke it. Unit tests would likely have prevented the code change

from reaching the Integration Build.

Missing Unit Tests can also arise during test-driven development when devel-

opers get ahead of themselves and write some code without having a failing test

to guide them.

Possible Solution

The trite answer is to write more unit tests. Of course, this is easier said than

done, and it isn’t always effective. Doing true test-driven development is the

best way to avoid having Missing Unit Tests without writing unnecessary tests

merely to get the test count up.

Cause: Untested Code

Symptoms

We may just “know” that some piece of code in the SUT is not being exercised

by any tests. Perhaps we have never seen that code execute, or perhaps we used

code coverage tools to prove this fact beyond a doubt. In the following example,

how can we test that when timeProvider throws an exception, this exception is

handled correctly?

Production Bugs

Production

Bugs

272

Chapter 17 Project Smells

public String getCurrentTimeAsHtmlFragment()

throws TimeProviderEx {

Calendar currentTime;

try {

currentTime = getTimeProvider().getTime();

} catch (Exception e) {

return e.getMessage();

}

// etc.

Root Cause

The most common cause of Untested Code is that the SUT includes code paths

that react to particular ways that a depended-on component (DOC) behaves

and we haven’t found a way to exercise those paths. Typically, the DOC is

being called synchronously and either returns certain values or throws excep-

tions. During normal testing, only a subset of the possible equivalence classes

of indirect inputs are actually encountered.

Another common cause of Untested Code is incompleteness of the test suite

caused by incomplete characterization of the functionality exposed via the

SUT’s interface.

Possible Solution

If the Untested Code is caused by an inability to control the indirect inputs of

the SUT, the most common solution is to use a Test Stub (page 529) to feed the

various kinds of indirect inputs into the SUT to cover all the code paths. Other-

wise, it may be sufﬁ cient to conﬁ gure the DOC to cause it to return the various

indirect inputs required to fully test the SUT.

Cause: Untested Requirement

Symptoms

We may just “know” that some piece of functionality is not being tested. Alter-

natively, we may be trying to test a piece of software but cannot see any visible

functionality that can be tested via the public interface of the software. All the

tests we have written pass, however.

When doing test-driven development, we know we need to add some

code to handle a requirement. However, we cannot ﬁ nd a way to express the

need for code to log the action in a Fully Automated Test (see page 26) such

as this:

Production

Bugs

273

public void testRemoveFlight() throws Exception {

// set up

FlightDto expectedFlightDto = createARegisteredFlight();

FlightManagementFacade facade =

new FlightManagementFacadeImpl();

// exercise

facade.removeFlight(expectedFlightDto.getFlightNumber());

// verify

assertFalse("ﬂight should not exist after being removed",

facade.ﬂightExists( expectedFlightDto.

getFlightNumber()));

}

Note that this test does not verify that the correct logging action has been done.

It will pass regardless of whether the logging was implemented correctly—or

even at all. Here’s the code that this test is verifying, complete with the indirect

output of the SUT that has not been implemented correctly:

public void removeFlight(BigDecimal ﬂightNumber)

throws FlightBookingException {

System.out.println(" removeFlight("+ﬂightNumber+")");

dataAccess.removeFlight(ﬂightNumber);

logMessage("CreateFlight", ﬂightNumber); // Bug!

}

If we plan to depend on the information captured by logMessage when maintain-

ing the application in production, how can we ensure that it is correct? Clearly,

it is desirable to have automated tests verify this functionality.

Impact

Part of the required behavior of the SUT could be accidentally disabled without

causing any tests to fail. Buggy software could be delivered to the customer. The

fear of introducing bugs could discourage ruthless refactoring or deletion of

code suspected to be unneeded (i.e., dead code).

Root Cause

The most common cause of Untested Requirements is that the SUT includes

behavior that is not visible through its public interface. It may have expected

“side effects” that cannot be observed directly by the test (such as writing out

a ﬁ le or record or calling a method on another object or component)—in other

words, it may have indirect outputs.

When the SUT is an entire application, the Untested Requirement may be a

result of not having a full suite of customer tests that verify all aspects of the

visible behavior of the SUT.

Production Bugs

Production

Bugs

274

Chapter 17 Project Smells

Production

Bugs

Possible Solution

If the problem is missing customer tests, we need to write at least enough cus-

tomer tests to ensure that all components are integrated properly. This may

require improving the design-for-testability of the application by separating the

presentation layer from the business logic layer.

When we have indirect outputs that we need to verify, we can do Behavior

Veriﬁ cation (page 468) through the use of Mock Objects (page 544). Testing of

indirect outputs is covered in Chapter 11, Using Test Doubles.

Cause: Neverfail Test

Symptoms

We may just “know” that some piece of functionality is not working, even

though the tests for that functionality pass. When doing test-driven develop-

ment, we have added a test for functionality we have not yet written but we

cannot get the test to fail.

Impact

If a test won’t fail even when the code to implement the functionality doesn’t

exist, how useful is it for Defect Localization (see page 22)? Not very!

Root Cause

This problem can be caused by improperly coded assertions such as assertTrue-

(aVariable, true)

instead of assertEquals(aVariable, true) or just assertTrue(aVariable).

Another cause is more sinister: When we have asynchronous tests, failures thrown

in the other thread or process may not be seen or reported by the Test Runner.

Possible Solution

We can implement cross-thread failure detection mechanisms to ensure that

asynchronous tests do, indeed, fail. An even better solution is to refactor the

code to support a Humble Executable (see Humble Object on page 695).

PART III

The Patterns

The

Patterns

This page intentionally left blank

277

Chapter 18

Test Strategy Patterns

Patterns in This Chapter

Test Automation Strategy

Recorded Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278

Scripted Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285

Data-Driven Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288

Test Automation Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . 298

Test Fixture Strategy

Minimal Fixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302

Standard Fixture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305

Fresh Fixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

Shared Fixture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317

SUT Interaction Strategy

Back Door Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

Layer Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337

Test Strategy

Patterns