Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

Chapter 1 A Brief Tour

What’s Next?

This whirlwind tour of the most important goals, principles, patterns, and

smells is just a brief introduction to test automation. Chapters 2 through 14

give a more detailed overview of each area touched upon here. If you have

already spotted some patterns or smells you want to learn more about, you can

certainly proceed directly to the detailed descriptions in Parts II and III. Other-

wise, your next step is to delve into the subsequent narratives, which provide a

somewhat more in-depth examination of these patterns and the alternatives to

them. First up is Chapter 2, Test Smells, which describes some common “test

smells” that motivate much of the refactoring we do on our tests.

Chapter 2

Test Smells

About This Chapter

Chapter 1, A Brief Tour, provided a very quick introduction to the core patterns

and smells covered in this book. This chapter provides a more detailed examina-

tion of the “test smells” we are likely to encounter on our projects. We explore

the basic concept of test smells ﬁ rst, and then move on to investigate the smells

in three broad categories: test code smells, automated test behavior smells, and

project smells related to automated testing.

An Introduction to Test Smells

In his book Refactoring: Improving the Design of Existing Code, Martin Fowler

documented a number of ways that the design of code can be changed without

actually changing what the code does. The motivation for this refactoring was

the identiﬁ cation of “bad smells” that frequently occur in object-oriented code.

These code smells were described in a chapter coauthored by Kent Beck that

started with the famous quote from Grandma Beck: “If it stinks, change it.” The

context of this quote was the question, “How do you know you need to change

a baby’s diaper?” And so a new term was added to the programmer’s lexicon.

The code smells described in Refactoring focused on problems commonly

found in production code. Many of us had long suspected that there were

smells unique to automated test scripts. At XP2001, the paper “Refactoring

Test Code” [RTC]

conﬁ rmed these suspicions by identifying a number of “bad

smells” that occur speciﬁ cally in test code. The authors also recommended a set

of refactorings that can be applied to the tests to remove the noxious smells.

This chapter provides an overview of these test smells. More detailed ex-

amples of each test smell can be found in the reference section.

What’s a Test Smell?

A smell is a symptom of a problem. A smell doesn’t necessarily tell us what is

wrong, because a particular smell may originate from any of several sources.

Most of the smells in this book have several different named causes; some causes

even appear under several smells. That’s because a root cause may reveal itself

through several different symptoms (i.e., smells).

Not all problems are considered smells, and some problems may even be the

root cause of several smells. The “Occam’s razor” test for deciding whether

something really is a smell (versus just a problem) is the “sniffability test.” That

is, the smell must grab us by the nose and say, “Something is wrong here.” As

discussed in the next section, I have classiﬁ ed the smells based on the kinds of

symptoms they exhibit (how they “grab us by the nose”).

Based on the “sniffability” criteria, I have demoted some of the test smells

listed in prior papers and articles to “cause” status. I have mostly left their

names unchanged so that we can still refer to them when talking about a par-

ticular side effect of applying a pattern. In this case, it is more appropriate to

refer directly to the cause rather than to the more general but sniffable smell.

Kinds of Test Smells

Over the years we have discovered that there are at least two different kinds of

smells: code smells, which must be recognized when looking at code, and behav-

ior smells, which affect the outcome of tests as they execute.

Code smells are coding-level anti-patterns that a developer, tester, or coach

may notice while reading or writing test code. That is, the code just doesn’t look

quite right or doesn’t communicate its intent very clearly. Code smells must ﬁ rst

be recognized before we can take any action, and the need for action may not be

equally obvious to everyone. Code smells apply to all kinds of tests, including both

Scripted Tests (page 285) and Recorded Tests (page 278). They become particu-

larly relevant for Recorded Tests when we must maintain the recorded code. Un-

fortunately, most Recorded Tests suffer from Obscure Tests (page 186), because

they are recorded by a tool that doesn’t know what is relevant to the human reader.

Behavior smells, by contrast, are much more difﬁ cult to ignore because they

cause tests to fail (or not compile at all) at the most inopportune times, such as

when we are trying to integrate our code into a crucial build; we are forced to

unearth the problems before we can “make the bar green.” Like code smells,

behavior smells are relevant to both Scripted Tests and Recorded Tests.

Chapter 2 Test Smells

Developers typically notice code and behavior smells when they automate,

maintain, and run tests. More recently, we have identiﬁ ed a third kind of

smell—a smell that is usually noticed by the project manager or the customer,

who does not look at the test code or run the tests. These project smells are in-

dicators of the overall health of a project.

What to Do about Smells?

Some smells are inevitable simply because they take too much effort to elimi-

nate. The important thing is that we are aware of the smells and know what

causes them. We can then make a conscious decision about which ones we must

address to keep the project running efﬁ ciently.

The decision of which smells must be eliminated comes down to the

balance between cost and beneﬁ t. Some smells are harder to stamp out than

others; some smells cause more grief than others. We need to eradicate those

smells that cause us the most grief because they will keep us from being suc-

cessful. That being said, many smells can be avoided by selecting a sound test

automation strategy and by following good test automation coding standards.

While we carefully delineated the various types of smells, it is important to

note that very often we will observe symptoms of each kind of smell at the

same time. Project smells, for example, are the project-level symptoms of some

underlying cause. That cause may show up as a behavior smell but ultimately

there is probably an underlying code smell that is the root cause of the problem.

The good news: We have three different ways to identify a problem. The bad

news: It is easy to focus on the symptom at one level and to try to solve that

problem directly without understanding the root cause.

A very effective technique for identifying the root cause is the “Five Why’s”

[TPS]. First, we ask why something is occurring. Once we have identiﬁ ed the

factors that led to it, we next ask why each of those factors occurred. We repeat

this process until no new information is forthcoming. In practice, asking why

ﬁ ve times is usually enough—hence the name “Five Why’s.”

In the rest of this chapter, we will look at the test-related smells that we

are most likely to encounter on our projects. We will begin with the project

smells, and then work our way down to the behavior smells and code smells

that cause them.

This practice is also called “root cause analysis” or “peeling the onion” in some circles.

An Introduction To Test Smells

Chapter 2 Test Smells

A Catalog of Smells

Now that we have a better understanding of test smells and their role in projects

that use automated testing, let’s look at some smells. Based on the “sniffability”

criteria outlined earlier, this section focuses on introducing the smells. Discus-

sions of their causes and the individual smell descriptions appear in Part II of

this book.

The Project Smells

Project smells are symptoms that something has gone wrong on the project. Their

root cause is likely to be one or more of the code or behavior smells. Because proj-

ect managers rarely run or write tests, however, project smells are likely to be the

ﬁ rst hint they get that something may be less than perfect in test automation land.

Project managers focus most on functionality, quality, resources, and cost.

For this reason, the project-level smells tend to cluster around these issues. The

most obvious metric a project manager is likely to encounter as a smell is the

quality of the software as measured in defects found in formal testing or by

users/customers. If the number of Production Bugs (page 268) is higher than

expected, the project manager must ask, “Why are all of these bugs getting

through our safety net of automated tests?”

The project manager may be monitoring the number of times the daily in-

tegration build fails as a way of getting an early indication of software quality

and adherence to the team’s development process. The manager may become

worried if the build fails too frequently, and especially if it takes more than a

few minutes to ﬁ x the build. Root cause analysis of the failures may indicate

that many of the test failures are not the result of buggy software but rather

derive from Buggy Tests (page 260). This is an example in which the tests cry

“Wolf!” and consume a lot of resources as part of their correction, but do not

actually increase the quality of the production code.

Buggy Tests are just one contributor to the more general problem of High

Test Maintenance Cost (page 265), which can severely affect the productivity

of the team if not addressed quickly. If the tests need to be modiﬁ ed too often

(e.g., every time the SUT is modiﬁ ed) or if the cost of modifying tests is too high

due to Obscure Tests, the project manager may decide that the effort and ex-

pense being directed toward writing the automated tests would be better spent

on writing more production code or doing manual testing. At this point, the

manager is likely to tell the developers to stop writing tests.

Alternatively, the project manager may decide that the Production Bugs are

caused by Developers Not Writing Tests (page 263). This pronouncement is

likely to come during a process retrospective or as part of a root cause analysis

session. Developers Not Writing Tests may be caused by an overly aggressive

development schedule, supervisors who tell developers not to “waste time writ-

ing tests,” or developers who do not have the skills to write tests. Other poten-

tial causes might include an imposed design that is not conducive to testing or

a test environment that leads to Fragile Tests (page 239). Finally, this problem

could result from Lost Tests (see Production Bugs)—tests that exist but are not

included in the AllTests Suite (see Named Test Suite on page 592) used by devel-

opers during check-in or by the automated build tool.

The Behavior Smells

Behavior smells are encountered when we compile or run tests. We don’t have

to be particularly observant to notice them, as these smells will take the form of

compile errors or test failures.

The most common behavior smell is Fragile Tests. It arises when tests that

once passed begin failing for some reason. The Fragile Test problem has given

test automation a bad name in many circles, especially when commercial

“record and playback” test tools fail to deliver on their promise of easy test

automation. Once recorded, these tests are very susceptible to breakage. Often

the only remedy is to rerecord them because the test recordings are difﬁ cult to

understand or modify by hand.

The root causes of Fragile Tests can be classiﬁ ed into four broad categories:

• Interface Sensitivity (see Fragile Test) occurs when tests are broken by

changes to the test programming API or the user interface used to au-

tomate the tests. Commercial Record and Playback Test (see Recorded

Test) tools typically interact with the system via the user interface. Even

minor changes to the interface can cause tests to fail, even in circum-

stances in which a human user would say that the test should still pass.

It can be hard enough to get project managers to buy into letting developers write

automated tests. It is crucial that we don’t squander this opportunity by being sloppy

or inefﬁ cient. The need for this balancing act is, in a nutshell, why I started writing this

book: to help developers succeed and avoid giving the pessimistic project manager an

excuse for calling a halt to automated unit testing.

A Catalog of Smells

Chapter 2 Test Smells

• Behavior Sensitivity (see Fragile Test) occurs when tests are broken by

changes to the behavior of the SUT. This may seem like a “no-brainer”

(of course, the tests should break if we change the SUT!) but the issue

is that only a few tests should be broken by any one change. If many or

most of the tests break, we have a problem.

• Data Sensitivity (see Fragile Test) occurs when tests are broken by

changes to the data already in the SUT. This issue is particularly a

problem for applications that use databases. Data Sensitivity is a spe-

cial case of Context Sensitivity (see Fragile Test) where the context in

question is the database.

• Context Sensitivity occurs when tests are broken by differences in the

environment surrounding the SUT. The most common example is when

tests depend on the time or date, but this problem can also arise when

tests rely on the state of devices such as servers, printers, or monitors.

Data Sensitivity and Context Sensitivity are examples of a special kind of Frag-

ile Test, known as a Fragile Fixture, in which changes to a commonly used test

ﬁ xture cause multiple existing tests to fail. This scenario increases the cost of

extending the Standard Fixture (page 305) to support new tests and, in turn,

discourages good test coverage. Although Fragile Fixture’s root cause is poor

test design, the problem actually appears when the ﬁ xture is changed rather

than when the SUT is changed.

Most agile projects use some form of daily or continuous integration that

includes two steps: compiling the latest version of the code and running all

of the automated tests against the newly compiled build. Assertion Rou-

lette (page 224) can make it difﬁ cult to determine how and why tests failed

during the integration build because the failure log does not include sufﬁ -

cient information to clearly identify which assertion failed. Troubleshooting

of the build failures may proceed slowly, because the failure must be repro-

duced in the development environment before we can speculate on the cause

of the failure.

A common cause of grief is tests that fail for no apparent reason. That is,

neither the tests nor the production code has been modiﬁ ed, yet the tests sud-

denly begin failing. When we try to reproduce these results in the development

environment, the tests may or may not fail. These Erratic Tests (page 228) are

both very annoying and time-consuming to ﬁ x, because they have numerous

possible causes. A few are listed here:

• Interacting Tests arise when several tests use a Shared Fixture (page 317).

They make it hard to run tests individually or to run several test suites as

part of a larger Suite of Suites (see Test Suite Object on page 387). They

can also cause cascading failures (where a single test failure leaves the

Shared Fixture in a state that causes many other tests to fail).

• Test Run Wars occur when several Test Runners (page 377) run tests

against a Shared Fixture at the same time. They invariably happen at

the worst possible time, such as when you are trying to ﬁ x the last few

bugs before a release.

• Unrepeatable Tests provide a different result between the ﬁ rst and

subsequent test runs. They may force the test automater to perform a

Manual Intervention (page 250) between test runs.

Another productivity-sapping smell is Frequent Debugging (page 248). Auto-

mated unit tests should obviate the need to use a debugger in all but rare cases,

because the set of tests that are failing should make it obvious why the failure is

occurring. Frequent Debugging is a sign that the unit tests are lacking in cover-

age or are trying to test too much functionality at once.

The real value of having Fully Automated Tests (page 26) is being able to run

them frequently. Agile developers who are doing test-driven development often

run (at least a subset of) the tests every few minutes. This behavior should be

encouraged because it shortens the feedback loop, thereby reducing the cost of

any defects introduced into the code. When tests require Manual Intervention

each time they are run, developers tend to run the tests less frequently. This

practice increases the cost of ﬁ nding all defects introduced since the tests were

last run, because more changes will have been made to the software since it was

last tested.

Another smell that has the same net impact on productivity is Slow

Tests (page 253). When tests take more than approximately 30 seconds to run,

developers stop running them after every individual code change, instead wait-

ing for a “logical time” to run them—for example, before a coffee break, lunch,

or a meeting. This delayed feedback results in a loss of “ﬂ ow” and increases

the time between when a defect is introduced and when it is identiﬁ ed by a test.

The most frequently used solution to Slow Tests is also the most problematic;

a Shared Fixture can result in many behavior smells and should be the solution

of last resort.

A Catalog of Smells

Chapter 2 Test Smells

The Code Smells

Code smells are the “classic” bad smells that were ﬁ rst described by Martin

Fowler in Refactoring [Ref]. Indeed, most of the smells identiﬁ ed by Fowler are

code smells. These smells must be recognized by test automaters as they main-

tain test code. Although code smells typically affect maintenance cost of tests,

they may also be early warning signs of behavior smells to follow.

When reading tests, a fairly obvious—albeit often overlooked—smell is

Obscure Test. It can take many forms, but all versions have the same impact: It

becomes difﬁ cult to tell what the test is trying to do, because the test does not

Communicate Intent (page 41). This ambiguity increases the cost of test main-

tenance and can lead to Buggy Tests when a test maintainer makes the wrong

change to the test.

A related smell is Conditional Test Logic (page 200). Tests should be simple,

linear sequences of statements. When tests have multiple execution paths, we

cannot be sure exactly how the test will execute in a speciﬁ c case.

Hard-Coded Test Data (see Obscure Test) can be insidious for several rea-

sons. First, it makes tests more problematic to understand: We need to look at

each value and guess whether it is related to any of the other values to under-

stand how the SUT is supposed to behave. Second, it creates challenges when

we are testing a SUT that includes a database. Hard-Coded Test Data can lead

to Erratic Tests (if tests happen to use the same database key) or Fragile Fix-

tures (if the values refer to records in the database that have been changed).

Hard-to-Test Code (page 209) may be a contributing factor to a number of

other code and behavior smells. This problem is most obvious to the person

who is writing a test and cannot ﬁ nd a way to set up the ﬁ xture, exercise the

SUT, or verify the expected outcome. The test automater may then be forced

to test more software (a larger SUT consisting of many classes) than he or she

would like. When reading a test, Hard-to-Test Code tends to show up as an

Obscure Test because of the hoops the test automater had to jump through to

interact with the SUT.

Test Code Duplication (page 213) is a poor practice because it increases the

cost of maintaining tests. We have more test code to maintain and that code

is more challenging to maintain because it often coincides with an Obscure

Test. Duplication often arises when the automated tester clones tests and does

not put enough thought into how to reuse test logic intelligently.

As testing

needs emerge, it is important that the test automater factor out commonly used

sequences of statements into Test Utility Methods (page 599) that can be reused

Note that I said “reuse test logic” and not “reuse Test Methods.”

by various Test Methods (page 348).

This practice reduces the maintenance

cost of tests in several ways.

Test Logic in Production (page 217) is undesirable because there is no way

to ensure that it will not run accidentally.

It also makes the production code

larger and more complicated. Finally, this error may cause other software com-

ponents or libraries to be included in the executable.

What’s Next?

In this chapter, we saw a plethora of things that can go wrong when automating

tests. Chapter 3, Goals of Test Automation, describes the goals we need to keep

in mind while automating tests so that we can have an effective test automation

experience. That understanding will prepare us to look at the principles that will

help us steer clear of many of the problems described in this chapter.

It is equally important that we do not reuse Test Methods, as that practice results in

Flexible Tests (see Conditional Test Logic).

See the sidebar on Ariane (page 218) for a cautionary tale.

What’s Next