Meszaros G. xUnit Test Patterns Refactoring Test Code
Подождите немного. Документ загружается.
228
Chapter 16 Behavior Smells
Erratic Test
One or more tests behave erratically; sometimes they pass
and sometimes they fail.
Symptoms
We have one or more tests that run but give different results depending on when
they are run and who is running them. In some cases, the Erratic Test will con-
sistently give the same results when run by one developer but fail when run by
someone else or in a different environment. In other cases, the Erratic Test will
give different results when run from the same Test Runner (page 377).
Impact
We may be tempted to remove the failing test from the suite to “keep the bar
green” but this would result in an (intentional) Lost Test (see Production Bugs
on page 268). If we choose to keep the Erratic Test in the test suite despite the
failures, the known failure may obscure other problems, such as another issue
detected by the same tests. Just having a test fail can cause us to miss additional
failures because it is much easier to see the change from a green bar to a red bar
than to notice that two tests are failing instead of just the one we expected.
Troubleshooting Advice
Erratic Tests can be challenging to troubleshoot because so many potential causes
exist. If the cause cannot be easily determined, it may be necessary to collect data
systematically over a period of time. Where (in which environments) did the
tests pass, and where did they fail? Were all the tests being run or just a subset
of them? Did any change in behavior occur when the test suite was run several
times in a row? Did any change in behavior occur when it was run from several
Test Runners at the same time?
Once we have some data, it should be easier to match up the observed symp-
toms with those listed for each of the potential causes and to narrow the list of
possibilities to a handful of candidates. Then we can collect some more data
focusing on differences in symptoms between the possible causes. Figure 16.1
summarizes the process for determining which cause of an Erratic Test we are
dealing with.
Erratic Test
229
Figure 16.1 Troubleshooting an Erratic Test.
Causes
Tests may behave erratically for a number of reasons. The underlying cause can
usually be determined through some persistent sleuthing by paying attention to
patterns regarding how and when the tests fail. Some of the causes are common
enough to warrant giving them names and specifi c advice for rectifying them.
Cause: Interacting Tests
Tests depend on other tests in some way. Note that Interacting Test Suites and
Lonely Test are specifi c variations of Interacting Tests.
Symptoms
A test that works by itself suddenly fails in the following circumstances:
• Another test is added to (or removed from) the suite.
• Another test in the suite fails (or starts to pass).
• The test (or another test) is renamed or moved in the source fi le.
• A new version of the Test Runner is installed.
Root Cause
Interacting Tests usually arise when tests use a Shared Fixture (page 317), with
one test depending in some way on the outcome of another test. The cause of
Interacting Tests can be described from two perspectives:
Results
Vary for Tests
vs. Suites?
Different
Results Every
Run?
No
Probably
Unrepeatable
Te st
Ye s
Gets
Worse with
Time?
No
Probably
Interacting
Tests or Suites
Happens
When Test Run
Alone?
No
No
Probably
Lonely Test
Ye s
Ye s
Probably
Resource
Leakage
Ye s
Probably
Resource
Optimism
Only with
Multiple Test
Runners?
No
Probably Test
Run War
Ye s
Different
Results for First
Run?
No
Probably Non-
Deterministic
Te st
Ye s
Results
Vary by
Location?
No
Ye s
Hire an
xUnit
Expert!
Results
Vary for Tests
vs. Suites?
Different
Results Every
Run?
No
Probably
Unrepeatable
Te st
Ye s
Gets
Worse with
Time?
No
Probably
Interacting
Tests or Suites
Happens
When Test Run
Alone?
No
No
Probably
Lonely Test
Ye s
Ye s
Probably
Resource
Leakage
Ye s
Probably
Resource
Optimism
Only with
Multiple Test
Runners?
No
Probably Test
Run War
Ye s
Different
Results for First
Run?
No
Probably Non-
Deterministic
Te st
Ye s
Results
Vary by
Location?
No
Ye s
Hire an
xUnit
Expert!
Erratic Test
Erratic Test
230
Chapter 16 Behavior Smells
• The mechanism of interaction
• The reason for interaction
The mechanism for interaction could be something blatantly obvious—for
example, testing an SUT that includes a database—or it could be more subtle.
Anything that outlives the lifetime of the test can lead to interactions; static
variables can be depended on to cause Interacting Tests and, therefore, should
be avoided in both the SUT and the Test Automation Framework (page 298)!
See the sidebar “There’s Always an Exception” on page 384 for an exam-
ple of the latter problem. Singletons [GOF] and Registries [PEAA] are good
examples of things to avoid in the SUT if at all possible. If we must use them,
it is best to include a mechanism to reinitialize their static variables at the
beginning of each test.
Tests may interact for a number of reasons, either by design or by accident:
• Depending on the fi xture constructed by the fi xture setup phase of
another test
• Depending on the changes made to the SUT during the exercise SUT
phase of another test
• A collision caused by some mutually exclusive action (which may be
either of the problems mentioned above) between two tests run in the
same test run
The dependencies may suddenly cease to be satisfi ed if the depended-on test
• Is removed from the suite,
• Is modifi ed to no longer change the state of the SUT,
• Fails in its attempt to change the state of the SUT, or
• Is run after the test in question (because it was renamed or moved to a
different Testcase Class; see page 373).
Similarly, collisions may start occurring when the colliding test is
• Added to the suite,
• Passes for the fi rst time, or
• Runs before the dependent test.
In many of these cases, multiple tests will fail. Some of the tests may fail for a
good reason—namely, the SUT is not doing what it is supposed to do. Depen-
dent tests may fail for the wrong reason—because they were coded to depend
Erratic Test
231
on other tests’ success. As a result, they may be giving a “false-positive” (false-
failure) indication.
In general, depending on the order of test execution is not a wise approach
because of the problems described above. Most variants of the xUnit frame-
work do not make any guarantees about the order of test execution within a
test suite. (TestNG, however, promotes interdependencies between tests by pro-
viding features to manage the dependencies.)
Possible Solution
Using a Fresh Fixture (page 311) is the preferred solution for Interacting Tests; it
is almost guaranteed to solve the problem. If we must use a Shared Fixture, we
should consider using an Immutable Shared Fixture (see Shared Fixture) to pre-
vent the tests from interacting with one another through changes in the fi xture
by creating from scratch those parts of the fi xture that they intend to modify.
If an unsatisfi ed dependency arises because another test does not create
the expected objects or database data, we should consider using Lazy Setup
(page 435) to create the objects or data in both tests. This approach ensures
that the fi rst test to execute creates the objects or data for both tests. We can
put the fi xture setup code into a Creation Method (page 415) to avoid Test
Code Duplication (page 213). If the tests are on different Testcase Classes, we
can move the fi xture setup code to a Test Helper (page
643).
Sometimes the collision may be caused by objects or database data that are
created in our test but not cleaned up afterward. In such a case, we should con-
sider implementing Automated Fixture Teardown (see Automated Teardown on
page 503) to remove them safely and effi ciently.
A quick way to fi nd out whether any tests depend on one another is to run
the tests in a different order than the normal order. Running the entire test
suite in reverse order, for example, would do the trick nicely. Doing so regularly
would help avoid accidental introduction of Interacting Tests.
Cause: Interacting Test Suites
In this special case of Interacting Tests, the tests are in different test suites.
Symptoms
A test passes when it is run in its own test suite but fails when it is run within a
Suite of Suites (see Test Suite Object on page 387).
Suite1.run()--> Green
Suite2.run()--> Green
Suite(Suite1,Suite2).run()--> Test C in Suite2 fails
Erratic Test
Erratic Test
232
Chapter 16 Behavior Smells
Root Cause
Interacting Test Suites usually occur when tests in separate test suites try to cre-
ate the same resource. When they are run in the same suite, the fi rst one succeeds
but the second one fails while trying to create the resource.
The nature of the problem may be obvious just by looking at the test failure
or by reading the failed Test Method (page 348). If it is not, we can try remov-
ing other tests from the (nonfailing) test suite, one by one. When the failure
stops occurring, we simply examine the last test we removed for behaviors that
might cause the interactions with the other (failing) test. In particular, we need
to look at anything that might involve a Shared Fixture, including all places
where class variables are initialized. These locations may be within the Test
Method itself, within a setUp method, or in any Test Utility Methods (page 599)
that are called.
Warning: There may be more than one pair of tests interacting in the same test
suite! The interaction may also be caused by the Suite Fixture Setup (page 441)
or Setup Decorator (page 447) of several Testcase Classes clashing rather than
by a confl ict between the actual Test Methods!
Variants of xUnit that use Testcase Class Discovery (see Test Discovery on
page 393), such as NUnit, may appear to not use test suites. In reality, they
do—they just don’t expect the test automaters to use a Test Suite Factory (see
Test Enumeration on page 399) to identify the Test Suite Object to the Test
Runner.
Possible Solution
We could, of course, eliminate this problem entirely by using a Fresh Fixture.
If this solution isn’t within our scope, we could try using an Immutable Shared
Fixture to prevent the tests’ interaction.
If the problem is caused by leftover objects or database rows created by one
test that confl ict with the fi xture being created by a later test, we should con-
sider using Automated Teardown to eliminate the need to write error-prone
cleanup code.
Cause: Lonely Test
A Lonely Test is a special case of Interacting Tests. In this case, a test can be run
as part of a suite but cannot be run by itself because it depends on something in a
Shared Fixture that was created by another test (e.g., Chained Tests; see page 454)
or by suite-level fi xture setup logic (e.g., a Setup Decorator).
We can address this problem by converting the test to use a Fresh Fixture or
by adding Lazy Setup logic to the Lonely Test to allow it to run by itself.
Erratic Test
233
Cause: Resource Leakage
Tests or the SUT consume fi nite resources.
Symptoms
Tests run more and more slowly or start to fail suddenly. Reinitializing the Test
Runner, SUT, or Database Sandbox (page 650) clears up the problem—only to
have it reappear over time.
Root Cause
Tests or the SUT consume fi nite resources by allocating those resources and
failing to free them afterward. This practice may make the tests run more
slowly. Over time, all the resources are used up and tests that depend on them
start to fail.
This problem can be caused by one of two types of bugs:
• The SUT fails to clean up the resources properly. The sooner we detect
this behavior, the sooner we can track it down and fi x it.
• The tests themselves cause the resource leakage by allocating resources
as part of fi xture setup and failing to clean them up during fi xture
teardown.
Possible Solution
If the problem lies in the SUT, then the tests have done their job and we can fi x
the bug. If the tests are causing the resource leakage, then we must eliminate the
source of the leaks. If the leaks are caused by failure to clean up properly when
tests fail, we may need to ensure that all tests do Guaranteed In-line Teardown (see
In-line Teardown on page 509) or convert them to use Automated Teardown.
In general, it is a good idea to set the size of all resource pools to 1. This
choice will cause the tests to fail much sooner, allowing us to more quickly
determine which tests are causing the leak(s).
Cause: Resource Optimism
A test that depends on external resources has nondeterministic results depending
on when or where it is run.
Symptoms
A test passes when it is run in one environment and fails when it is run in
another environment.
Erratic Test
Erratic Test
234
Chapter 16 Behavior Smells
Root Cause
A resource that is available in one environment is not available in another
environment.
Possible Solution
If possible, we should convert the test to use a Fresh Fixture by creating the
resource as part of the test’s fi xture setup phase. This approach ensures that the
resource exists wherever it is run. It may necessitate the use of relative address-
ing of fi les to ensure that the specifi c location in the fi le system exists regardless
of where the SUT is executed.
If an external resource must be used, the resources should be stored in
the source code repository [SCM] so that all Test Runners run in the same en-
vironment.
Cause: Unrepeatable Test
A test behaves differently the fi rst time it is run compared with how it behaves
on subsequent test runs. In effect, it is interacting with itself across test runs.
Symptoms
Either a test passes the fi rst time it is run and fails on all subsequent runs, or it
fails the fi rst time and passes on all subsequent runs. Here’s an example of what
“Pass-Fail-Fail” might look like:
Suite.run()--> Green
Suite.run()--> Test C fails
Suite.run()--> Test C fails
User resets something
Suite.run()--> Green
Suite.run()--> Test C fails
Here’s an example of what “Fail-Pass-Pass” might look like:
Suite.run()--> Test C fails
Suite.run()--> Green
Suite.run()--> Green
User resets something
Suite.run()--> Test C fails
Suite.run()--> Green
Be forewarned that if our test suite contains several Unrepeatable Tests, we may
see results that look more like this:
Suite.run()--> Test C fails
Suite.run()--> Test X fails
Suite.run()--> Test X fails
Erratic Test
235
User resets something
Suite.run()--> Test C fails
Suite.run()--> Test X fails
Test C exhibits the “Fail-Pass-Pass” behavior, while test X exhibits the “Pass-
Fail-Fail” behavior at the same time. It is easy to miss this problem because we
see a red bar in each case; we notice the difference only if we look closely to see
which tests fail each time we run them.
Root Cause
The most common cause of an Unrepeatable Test is the use—either deliberate
or accidental—of a Shared Fixture. A test may be modifying the test fi xture such
that, during a subsequent run of the test suite, the fi xture is in a different state.
Although this problem most commonly occurs with a Prebuilt Fixture (see Shared
Fixture), the only true prerequisite is that the fi xture outlasts the test run.
The use of a Database Sandbox may isolate our tests from other developers’
tests but it won’t prevent the tests we run from colliding with themselves or
with other tests we run from the same Test Runner.
The use of Lazy Setup to initialize a fi xture holding class variable can result
in the test fi xture not being reinitialized on subsequent runs of the same test
suite. In effect, we are sharing the test fi xture between all runs started from the
same Test Runner.
Possible Solution
Because a persistent Shared Fixture is a prerequisite for an Unrepeatable Test,
we can eliminate the problem by using a Fresh Fixture for each test. To fully
isolate the tests, we must make sure that no shared resource, such as a Database
Sandbox, outlasts the lifetimes of the individual tests. One option is to replace
a database with a Fake Database (see Fake Object on page 551). If we must
work with a persistent data store, we should use Distinct Generated Values (see
Generated Value on page 723) for all database keys to ensure that we create
different objects for each test and test run. The other alternative is to implement
Automated Teardown to remove all newly created objects and rows safely and
effi ciently.
Cause: Test Run War
Test failures occur at random when several people are running tests
simultaneously.
Erratic Test
Erratic Test
236
Chapter 16 Behavior Smells
Symptoms
We are running tests that depend on some shared external resource such as a
database. From the perspective of a single person running tests, we might see
something like this:
Suite.run() --> Test 3 fails
Suite.run() --> Test 2 fails
Suite.run() --> All tests pass
Suite.run() --> Test 1 fails
Upon describing our problem to our teammates, we discover that they are
having the same problem at the same time. When only one of us runs tests, all
of the tests pass.
Impact
A Test Run War can be very frustrating because the probability of it occurring
increases the closer we get to a code cutoff deadline. This isn’t just Murphy’s law
kicking in: It really does happen more often at this point! We tend to commit
smaller changes at more frequent intervals as the deadline approaches (think
“last-minute bug fi xing”!). This, in turn, increases the likelihood that someone
else will be running the test suite at the same time, which itself increases the like-
lihood of test collisions between test runs occurring at the same time.
Root Cause
A Test Run War can happen only when we have a globally Shared Fixture that
various tests access and sometimes modify. This shared fi xture could be a fi le
that must be opened or read by either a test or the SUT, or it could consist of the
records in a test database.
Database contention can be caused by the following activities:
• Trying to update or delete a record while another test is also updating
the same record
• Trying to update or delete a record while another test has a read lock
(pessimistic locking) on the same record
File contention can be caused by an attempt to access a fi le that has already been
opened by another instance of the test running from a different Test Runner.
Possible Solution
Using a Fresh Fixture is the preferred solution for a Test Run War. An even sim-
pler solution is to give each Test Runner his or her own Database Sandbox. This
Erratic Test
237
should not involve making any changes to the tests but will completely eliminate
the possibility of a Test Run War. It will not, however, eliminate other sources of
Erratic Tests because the tests can still interact through the Shared Fixture (the
Database Sandbox). Another option is to switch to an Immutable Shared Fixture
by having each test create new objects whenever it plans to change those objects.
This approach does require changes to the Test Methods.
If the problem is caused by leftover objects or database rows created by one
test that pollutes the fi xture of a later test, another solution is using Automated
Teardown to clean up after each test safely and effi ciently. This measure, by
itself, is unlikely to completely eliminate a Test Run War but it might reduce its
frequency.
Cause: Nondeterministic Test
Test failures occur at random, even when only a single Test Runner is running
tests.
Symptoms
We are running tests and the results vary each time we run them, as shown here:
Suite.run() --> Test 3 fails
Suite.run() --> Test 3 crashes
Suite.run() --> All tests pass
Suite.run() --> Test 3 fails
After comparing notes with our teammates, we rule out a Test Run War either
because we are the only person running tests or because the test fi xture is not
shared between users or computers.
As with an Unrepeatable Test, having multiple Nondeterministic Tests in
the same test suite can make it more diffi cult to detect the failure/error pat-
tern: It looks like different tests are failing rather than a single test producing
different results.
Impact
Debugging Nondeterministic Tests can be very time-consuming and frustrating
because the code executes differently each time. Reproducing the failure can
be problematic, and characterizing exactly what causes the failure may require
many attempts. (Once the cause has been characterized, it is often a straight-
forward process to replace the random value with a value known to cause the
problem.)
Erratic Test
Erratic Test