Meszaros G. xUnit Test Patterns Refactoring Test Code
Подождите немного. Документ загружается.
This page intentionally left blank
259
Chapter 17
Project Smells
Smells in This Chapter
Buggy Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
Developers Not Writing Tests. . . . . . . . . . . . . . . . . . . . . . . . . . . 263
High Test Maintenance Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Production Bugs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268
Project
Smells
260
Chapter 17 Project Smells
Buggy Tests
Bugs are regularly found in the automated tests.
Fully Automated Tests (see page 26) are supposed to act as a “safety net”
for teams doing iterative development. But how can we be sure the safety net
actually works?
Buggy Tests is a project-level indication that all is not well with our auto-
mated tests.
Symptoms
A build fails, and a failed test is to blame. Upon closer inspection, we discover
that the code being testing works correctly, but the test indicated it was broken.
We encountered Production Bugs (page 268) despite having tests that verify
the specifi c scenario in which the bug was found. Root-cause analysis indicates
the test contains a bug that precluded catching the error in the production code.
Impact
Tests that give misleading results are dangerous! Tests that pass when they
shouldn’t (a false negative, as in “nothing wrong here”) give a false sense of
security. Tests that fail when they shouldn’t (a false positive) discredit the tests.
They are like the little boy who cried, “Wolf!”; after a few occurrences, we tend
to ignore them.
Causes
Buggy Tests can have many causes. Most of these problems also show up as
code or behavior smells. As project managers, we are unlikely to see these un-
derlying smells until we specifi cally look for them.
Cause: Fragile Test
Buggy Tests may just be project-level symptoms of a Fragile Test (page 239). For
false-positive test failures, a good place to start is the “four sensitivities”: Interface
Sensitivity (see Fragile Test), Behavior Sensitivity (see Fragile Test), Data Sensi-
tivity (see Fragile Test), and Context Sensitivity (see Fragile Test). Each of these
sensitivities could be the change that caused the test to fail. Removing the sensi-
tivities by using Test Doubles (page 522) and refactoring can be challenging but
ultimately it will make the tests much more dependable and cost-effective.
Buggy Tests
261
Cause: Obscure Test
A common cause of false-negative test results (tests that pass when they shouldn’t)
is an Obscure Test (page 186), which is diffi cult to get right—especially when
we are modifying existing tests that were broken by a change we made. Because
automated tests are hard to test, we don’t often verify that a modifi ed test still
catches all the bugs it was initially designed to trap. As long as we see a green
bar, we think we are “good to go.” In reality, we may have created a test that
never fails.
Obscure Tests are best addressed through refactoring of tests to focus on
the reader of the tests. The real goal is Tests as Documentation (see page 23)—
anything less will increase the likelihood of Buggy Tests.
Cause: Hard-to-Test Code
Another common cause of Buggy Tests, especially with “legacy software”
(i.e., any software that doesn’t have a complete suite of automated tests), is that the
design of the software is not conducive to automated testing. This Hard-to-Test
Code (page 209) may force us to use Indirect Testing (see Obscure Test), which
in turn may result in a Fragile Test.
The only way Hard-to-Test Code will become easy to test is if we refactor the
code to improve its testability. (This transformation is described in Chapter 6,
Test Automation Strategy, and Chapter 11, Using Test Doubles.) If this is not an
option, we may be able to reduce the amount of test code affected by a change
by applying SUT API Encapsulation (see Test Utility Method on page 599).
Troubleshooting Advice
When we have Buggy Tests, it is important to ask lots of questions. We must ask
the “fi ve why’s” [TPS] to get to the bottom of the problem—that is, we must
determine exactly which code and/or behavior smells are causing the Buggy
Tests and fi nd the root cause of each smell.
Solution Patterns
The solution depends very much on why the Buggy Tests occurred. Refer to the
underlying behavior and code smells for possible solutions.
As with all “project smells,” we should look for project-level causes. These
include not giving developers enough time to perform the following activities:
Buggy Tests
Buggy Tests
262
Chapter 17 Project Smells
• Learn to write the tests properly
• Refactor the legacy code to make test automation easier and more robust
• Write the tests fi rst
Failure to address these project-level causes guarantees that the problems will
recur in the near future.
Buggy Tests
263
Developers Not Writing Tests
Developers aren’t writing automated tests.
Symptoms
We hear that our developers aren’t writing tests. Or maybe we have observed
Production Bugs (page 268) and asked, “Why are so many bugs getting
through?”, only to be told, “Because we aren’t writing tests to cover that part
of the software.”
Impact
If the team isn’t writing automated tests for every piece of software “that could
possibly break,” it is mortgaging its future. The current pace of software develop-
ment will not be sustainable over the long haul because the system will be in test
debt. It will take longer and longer to add new functionality, and refactoring the
code to improve its design will be fraught with peril (so it will happen less and
less frequently). This problem marks the beginning of a trip down the proverbial
“slippery slope” to traditional paranoid, non-agile development. If that is where
we aspire to be, we should stay the course. Otherwise, it is time to take action.
Causes
Cause: Not Enough Time
Developers may have trouble writing tests in the time they are given to do
the development. This problem could be caused by an overly aggressive devel-
opment schedule or supervisors/team leaders who instruct developers, “Don’t
waste time writing tests.” Alternatively, developers may not have the skills
needed to write tests effi ciently and may not be allocated the time required to
work their way up the learning curve.
If time is what the developers need, managers need to adjust the proj-
ect schedule to give them that time. This extension need be only a temporary
adjustment while the developers learn the skills and test automation infrastructure
that will enable them to write the tests more quickly. In my experience, once
developers have internalized the process, they can write the tests and the code in
the same amount of time it once took them to write and debug just the code.
The time spent writing the tests is more than compensated for by the time not
spent in the debugger.
Developers Not Writing Tests
Developers
Not Writing
Tests
264
Chapter 17 Project Smells
Cause: Hard-to-Test Code
A common cause of Developers Not Writing Tests, especially with “legacy soft-
ware” (i.e., any software that doesn’t have a complete suite of automated tests), is
that the design of the software is not conducive to automated testing. This situa-
tion is described in more detail in its own smell, Hard-to-Test Code (page 209).
Cause: Wrong Test Automation Strategy
Another cause of Developers Not Writing Tests may be a test environment
or test automation strategy that leads to Fragile Tests (page 239) or Obscure
Tests (page 186) that take too long to write. We need to ask the “fi ve why’s”
[TPS] to fi nd the root causes. Then we can address those causes and get the ship
back on course.
Troubleshooting Advice
Project-level smells such as Developers Not Writing Tests are more likely to be
detected by a project manager, scrum master, or team leader than by a developer.
As managers, we may not know how to fi x the problem, but our awareness
and recognition of it is what matters. This unique perspective allows managers
to ask the development team questions about why they aren’t writing tests, in
which circumstances, and how long it takes to write tests when they do so. Then
managers can encourage and empower the developers to come up with ways of
addressing the root causes so that they write all the necessary tests.
Of course, managers must give the developers their full support in carrying
out whatever improvement plan they come up with. That support must include
enough time to learn the requisite skills and build or set up the necessary test
infrastructure. And managers shouldn’t expect things to turn around overnight.
They might set a process improvement goal for each iteration, such as “20%
reduction in code not tested” or “20% improvement in code coverage.” These
goals should be reasonable and at a high-enough level that they encourage the
right behavior, as opposed to just making the numbers look good. (A goal of 205
more tests written, for example, could be achieved without increasing the test
coverage one iota simply by splitting tests into smaller pieces or cloning tests.)
Developers
Not Writing
Tests
265
High Test Maintenance Cost
Too much effort is spent maintaining existing tests.
Test code needs to be maintained along with the production code it verifi es. As
an application evolves, we will likely have to revisit our tests on a regular basis
whenever we change the SUT classes to add new functionality or whenever we
refactor the tests to simplify those classes. High Test Maintenance Cost occurs
when the tests become overly diffi cult to understand and maintain.
Symptoms
Development of new functionality slows down. Every time we add some new
functionality, we need to make extensive changes to the existing tests. Develop-
ers or test automaters may tell the project manager or coach that they need a
“test refactoring/cleanup iteration.”
If we have been tracking the amount of time we spend writing the new tests
and modifying existing tests separately from the time we spend implementing
the code to make the tests pass, we notice that most of the time is spent modify-
ing the existing tests.
Most test maintainability issues are accompanied by other smells, such as the
following:
•A Fragile Test (page 239) indicates that tests are too closely coupled to
the SUT.
•AFragile Fixture (see Fragile Test) signals that too many tests depend
on the same fi xture design (Standard Fixture on page 305), which leads
to High Test Maintenance Cost.
•An Erratic Test (page 228) may be a sign that a Shared Fixture (page 317)
is causing our problem.
Impact
Team productivity drops signifi cantly because the tests take so much effort to main-
tain. Developers may be agitating to “cut and run” (remove the affected tests from
the test suites). While writing the production code is mandatory, maintaining the
tests is completely optional (at least to the uninformed). If nothing is done about this
problem, the entire test automation effort may be wasted when the team or manage-
ment decides that test automation just “doesn’t work” and abandons the tests.
High Test Maintenance Cost
High Test
Maintenance
Cost
266
Chapter 17 Project Smells
Causes
The root cause of High Test Maintenance Cost is failing to pay attention to
the principles described in Chapter 5, Principles of Test Automation. A more
immediate cause is often too much Test Code Duplication (page 213) and tests
that are too closely coupled to the API of the SUT.
Cause: Fragile Test
Tests that fail because minor changes were made to the SUT are called Fragile
Tests. They result in High Test Maintenance Cost because they need to be revis-
ited and “giggled” after all manner of minor changes that really shouldn’t affect
them.
The root cause of this failure can be any of the “four sensitivities”: Inter-
face Sensitivity (see Fragile Test), Behavior Sensitivity (see Fragile Test), Data
Sensitivity (see Fragile Test), and Context Sensitivity (see Fragile Test). We can
reduce the High Test Maintenance Cost by protecting the tests against as many
of these sensitivities as possible through the use of Test Doubles (page 522)
and by refactoring the system into smaller components and classes that can be
tested individually.
Cause: Obscure Test
Obscure Tests (page 186) are a major contributor to High Test Maintenance
Cost because they take longer to understand each time they are visited. When
they need to be modifi ed, they take more effort to adjust and are much less likely
to “work the fi rst time,” resulting in more debugging of tests. Obscure Tests are
also more likely to end up not catching conditions they were intended to detect,
which can lead to Buggy Tests (page 260).
Obscure Tests are best addressed by refactoring tests to focus on the reader
of the tests. The real goal is Tests as Documentation (see page 23)—anything
less will increase the likelihood of High Test Maintenance Cost.
Cause: Hard-to-Test Code
“Legacy software” (i.e., any software that doesn’t have a complete suite of auto-
mated tests) can be hard to test because we typically write the tests “last” (after
the software already exists). If the design of the software is not conducive to
automated testing, we may be forced to use Indirect Testing (see Obscure Test)
via awkward interfaces that involve a lot of accidental complexity; that effort
may result in Fragile Tests.
High Test
Maintenance
Cost
267
It will take both time and effort to refactor the code to improve its testability.
Nevertheless, that time and effort are well spent if they eliminate the High Test
Maintenance Cost. If refactoring is not an option, we may be able to reduce
the amount of test code affected by a change by doing SUT API Encapsulation
(see Test Utility Method on page 599) using Test Utility Methods. For example,
Creation Methods (page 415) encapsulate the constructors, thereby rendering
the tests less susceptible to changes in constructor signatures or semantics.
Troubleshooting Advice
As a project-level smell, High Test Maintenance Cost is as likely to be detected
by a project manager, scrum master, or team leader as by a developer. While
managers may not have the technical depth needed to troubleshoot and fi x the
problem, the fact that they become aware of it is what is important. This aware-
ness allows the manager to question the development team about how long it is
taking to maintain tests, how often test maintenance occurs, and why it is neces-
sary. Then the manager can challenge the developers to fi nd a better way—one
that won’t result in such High Test Maintenance Costs!
Of course, the developers will need the manager’s support to carry out
whatever improvement plan they come up with. That support must include
time to conduct the investigations (spikes), learning/training time, and time to
do the actual work. Managers can make time for this activity by having “test
refactoring stories,” adjusting the velocity to reduce the new functionality com-
mitted to the customer, or other means. Regardless of how managers carve out
this time, they must remember that if they don’t give the development team the
resources needed to fi x the problem now, the problem will simply get worse and
become even more challenging to fi x in the future when the team has twice as
many tests.
High Test Maintenance Cost
High Test
Maintenance
Cost