Meszaros G. xUnit Test Patterns Refactoring Test Code
Подождите немного. Документ загружается.
48
behavior of a SUT using metadata and we want to write tests that verify that
the metadata is set up correctly, we should not have to write code to do so. A
Data-Driven Test (page 288) would be much more appropriate in these circum-
stances.
What’s Next?
Previous chapters covered the common pitfalls (in the form of test smells) and
goals of test automation. This chapter made the value system we use while
choosing patterns explicit. In Chapter 6, Test Automation Strategy, we will
examine the “hard to change” decisions that we should try to get right early in
the project.
Chapter 5 Principles of Test Automation
49
Chapter 6
Test Automation Strategy
About This Chapter
In previous chapters, we saw some of the problems we might encounter with
test automation. In Chapter 5, Principles of Test Automation, we learned about
some of the principles we can apply to help address those problems. This chapter
gets a bit more concrete but still focuses at the 30,000-foot level. In the logical
sequence of things, test strategy comes before fi xture setup but is a somewhat
more advanced topic. If you are new to test automation using xUnit, you may
want to skip this chapter and come back after reading more about the basics
of xUnit in Chapter 7, xUnit Basics, and about fi xture setup and teardown in
Chapter 8, Transient Fixture Management, and subsequent chapters.
What’s Strategic?
As the story in the preface amply demonstrates, it is easy to get off on the wrong
foot. This is especially true when you lack experience in test automation and
when this testing strategy is adopted “bottom up.” If we catch the problems early
enough, the cost of refactoring the tests to eliminate the problems can be manage-
able. If, however, the problems are left to fester for too long or the wrong approach
is taken to address them, a very large amount of effort can be wasted. This is not
to suggest that we should follow a “big design upfront” (BDUF) approach to test
automation. BDUF is almost always the wrong answer. Rather, it is helpful to be
aware of the strategic decisions necessary and to make them “just in time” rather
than “much too late.” This chapter gives a “head’s up” about some of the strategic
issues we want to keep in mind so that we don’t get blindsided by them later.
What makes a decision “strategic”? A decision is strategic if it is “hard to
change.” That is, a strategic decision affects a large number of tests, especially
such that many or all the tests would need to be converted to a different approach
at the same time. Put another way, any decision that could cost a large amount of
effort to change is strategic.
Common strategic decisions include the following considerations:
• Which kinds of tests to automate?
• Which tools to use to automate them?
• How to manage the test fi xture?
• How to ensure that the system is easily tested and how the tests interact
with the SUT?
Each of these decisions can have far-reaching consequences, so they are best made
consciously, at the right time, and based on the best available information.
The strategies and more detailed patterns described in this book are equally
applicable regardless of the kind of Test Automation Framework (page 298)
we choose to use. Most of my experience is with xUnit, so it is the focus of this
book. But “don’t throw out the baby with the bath water”: If you fi nd yourself
using a different kind of Test Automation Framework, remember that most of
what you learn in regard to xUnit may still be applicable.
Which Kinds of Tests Should We Automate?
Roughly speaking, we can divide tests into the following two categories:
• Per-functionality tests (also known as functional tests) verify the behavior
of the SUT in response to a particular stimulus.
• Cross-functional tests verify various aspects of the system’s behavior
that cut across specifi c functionality.
Figure 6.1 shows these two basic kinds of tests as two columns, each of which is
further subdivided into more specifi c kinds of tests.
Per-Functionality Tests
Per-functionality tests verify the directly observable behavior of a piece of soft-
ware. The functionality can be business related (e.g., the principal use cases of
the system) or related to operational requirements (e.g., system maintenance
and specifi c fault-tolerance scenarios). Most of these requirements can also be
expressed as use cases, features, user stories, or test scenarios.
Per-functionality tests can be characterized by whether the functionality is
business (or user) facing and by the size of the SUT on which they operate.
50
Chapter 6 Test Automation Strategy
51
Figure 6.1 A summary of the kinds of tests we write and why. The left column
contains the tests we write that describe the functionality of the product at
various levels of granularity; we perform these tests to support development.
The right column contains tests that span specifi c chunks of functionality; we
execute these tests to critique the product. The bottom of each cell describes
what we are trying to communicate or verify.
Customer Tests
Customer tests verify the behavior of the entire system or application. They typi-
cally correspond to scenarios of one or more use cases, features, or user stories.
These tests often go by other names such as functional tests, acceptance tests, or
end-user tests. Although they may be automated by developers, their key char-
acteristic is that an end user should be able to recognize the behavior specifi ed
by the test even if the user cannot read the test representation.
Unit Tests
Unit tests verify the behavior of a single class or method that is a consequence
of a design decision. This behavior is typically not directly related to the require-
ments except when a key chunk of business logic is encapsulated within the
class or method in question. These tests are written by developers for their own
use; they help developers describe what “done looks like” by summarizing the
behavior of the unit in the form of tests.
Customer
Tests
Business Intent
(Executable Specification)
Usability
Testing
Exploratory
Testing
Unit
Tests
Developer Intent
(Design of the Code)
Property
Testing
Is it responsive,
secure, scalable?
Automated
Manual
Automated
Tool -
Technology
Facing
Business
Facing
Support
Development
Critique
Product
Automated
various
Manual
Automated
xUnit
Based
Special-Purpose
Tool-Based
Purpose of Tests
Component
Tests
Architect Intent
(Design of the System)
Is it pleasurable?
Is it self-consistent?
Manual
Manual
Automated
Automated
xUnit
Per Functionality
Cross-Functional
Kind of Behavior
Diagram adapted
from Mary
Poppendieck and
Brian Marick
Customer
Tests
Business Intent
(Executable Specification)
Usability
Testing
Exploratory
Testing
Unit
Tests
Developer Intent
(Design of the Code)
Property
Testing
Is it responsive,
secure, scalable?
Automated
Manual
Automated
Tool -
Technology
Facing
Business
Facing
Support
Development
Critique
Product
Automated
various
Manual
Automated
xUnit
Based
Special-Purpose
Tool-Based
Purpose of Tests
Component
Tests
Architect Intent
(Design of the System)
Is it pleasurable?
Is it self-consistent?
Manual
Manual
Automated
Automated
xUnit
Per Functionality
Cross-Functional
Kind of Behavior
Diagram adapted
from Mary
Poppendieck and
Brian Marick
Which Kinds of Tests Should We Automate?
52
Chapter 6 Test Automation Strategy
Component Tests
Component tests verify components consisting of groups of classes that collec-
tively provide some service. They fi t somewhere between unit tests and customer
tests in terms of the size of the SUT being verifi ed. Although some people call
these “integration tests” or “subsystem tests,” those terms can mean something
entirely different from “tests of a specifi c larger-grained subcomponent of the
overall system.”
Fault Insertion Tests
Fault insertion tests typically show up at all three levels of granularity within
these functional tests, with different kinds of faults being inserted at each level.
From a test automation strategy point of view, fault insertion is just another set
of tests at the unit and component test levels. Things get more interesting at the
whole-application level, however. Inserting faults here can be hard to automate
because it is challenging to automate insertion of the faults without replacing
parts of the application.
Cross-Functional Tests
Property Tests
Performance tests verify various “nonfunctional” (also known as “extra-functional”
or “cross-functional”) requirements of the system. These requirements are different
in that they span the various kinds of functionality. They often correspond to the
architectural “-ilities.” These kinds of tests include
• Response time tests
• Capacity tests
• Stress tests
From a test automation perspective, many of these tests must be automated (at
least partially) because human testers would have a hard time creating enough
load to verify the behavior under stress. While we can run the same test many
times in a row in xUnit, the xUnit framework is not particularly well suited to
automating performance tests.
One advantage of agile methods is that we can start running these kinds of
tests quite early in the project—as soon as the key components of the architecture
have been roughed in and the skeleton of the functionality is executable. The
same tests can then be run continuously throughout the project as new features
are added to the system skeleton.
53
Usability Tests
Usability tests verify “fi tness for purpose” by confi rming that real users can use
the software application to achieve the stated goals. These tests are very diffi cult
to automate because they require subjective assessment by people regarding how
easy it is to use the SUT. For this reason, usability tests are rarely automated and
will not be discussed further in this book.
Exploratory Testing
Exploratory testing is a way to determine whether the product is self-consistent.
The testers use the product, observe how it behaves, form hypotheses, design
tests to verify those hypotheses, and exercise the product with them. By its very
nature, exploratory testing cannot be automated, although automated tests can
be used to set up the SUT in preparation for doing exploratory testing.
Which Tools Do We Use to Automate Which Tests?
Choosing the right tool for the job is as important as having good skills with the
tools selected for use. A wide array of tools are available in the marketplace, and
it is easy to be seduced by the features of a particular tool. The choice of tool is
a strategic decision: Once we have invested a lot of time and effort in learning a
tool and automating many tests using that tool, it becomes much more diffi cult
to change to a different tool.
There are two fundamentally different approaches to automating tests
(Figure 6.2). The Recorded Test (page 278) approach involves the use of tools
that monitor our interactions with the SUT while we test it manually. This
information is then saved to a fi le or database and becomes the script for re-
playing this test against another (or even the same) version of the SUT. The
main problem with Recorded Tests is the level of granularity they record. Most
commercial tools record actions at the user interface (UI) element level, which
results in Fragile Tests (page 239).
The second approach to automating tests, Hand-Scripted Tests (see Scripted
Test on page 285), involves the hand-coding of test programs (“scripts”) that ex-
ercise the system. While xUnit is probably the most commonly used Test
Automation Framework for preparing Hand-Scripted Tests, they may be pre-
pared in other ways, including “batch” fi les, macro languages, and commercial or
open-source test tools. Some of the better-known open-source tools for preparing
Scripted Tests are Watir (test scripts coded in Ruby and run inside Internet Ex-
plorer), Canoo WebTest (tests scripted in XML and run using the WebTest tool),
Which Tools Do We Use to Automate Which Tests?
54
Chapter 6 Test Automation Strategy
and the ever-popular Fit (and its wiki-based sibling FitNesse). Some of these tools
even provide a test capture capability, thereby blurring the lines between Scripted
Tests and Recorded Tests.
Figure 6.2 A summary of the three dimensions of test automation choices.
The left side shows the two ways of interacting with the SUT. The bottom
edge enumerates how we create the test scripts. The front-to-back dimension
categorizes the different sizes of SUT we may choose to test.
Choosing which test automation tools to use is a large part of the test strategy
decision. A full survey of the different kinds of tools available is beyond the
scope of this book, but a somewhat more detailed treatment of the topic is avail-
able in [ARTRP]. The following sections summarize the information here to
provide an overview of the strengths and weaknesses of each approach.
Test Automation Ways and Means
Figure 6.3 depicts the decision-making possibilities as a matrix. In theory, there
are 2 × 2 × 3 possible combinations in this matrix, but it is possible to under-
stand the primary differences between the approaches by looking at the front
face of the cube. Some of the four quadrants are applicable to all levels of granu-
larity; others are primarily used for automating customer tests.
Scripted
system
component
unit
UI
API
Means of
Test – SUT
Interaction
Way of Captur ing Tests
SUT
Granular ity
Recorded
System
Component
Unit
Scripted
system
component
unit
UI
API
Means of
Test – SUT
Interaction
Way of Captur ing Tests
SUT
Granular ity
Recorded
System
Component
Unit
55
Figure 6.3 The choices on the front face of the cube. A more detailed look
at the front face of the cube in Figure 6.2 along with the advantages (+) and
disadvantages of each (–).
Upper-Right Quadrant: Modern xUnit
The upper-right quadrant of the front face of the cube is dominated by the xUnit
family of testing frameworks. These frameworks involve hand-scripting tests
that exercise the system at all three levels of granularity (system, component,
and unit) via internal interfaces. A good example is unit tests automated using
JUnit or NUnit.
Lower-Right Quadrant: Scripted UI Tests
This quadrant represents a variation on the “modern xUnit” approach, with the
most common examples being the use of HttpUnit, JFCUnit, Watir, or similar
tools to hand-script tests using the UI. It is also possible to hand-script tests using
commercial Recorded Test tools such as QTP. These approaches all reside within
the lower-right quadrant at various levels of SUT granularity. For example, when
used for customer tests, these tools would perform at the system test level of
granularity. They could also be used to test just the UI component of the system
(or possibly even some UI units such as custom widgets), although this effort
would require stubbing out the actual system behind the UI.
Lower-Left Quadrant: Robot User
The “robot user” quadrant focuses on recording tests that interact with the
system via the UI. Most commercial test automation tools follow this approach.
It applies primarily at the “whole system” granularity but, like scripted UI Tests,
Built -in
R&PB
Modern
XUnit
Robot
User
Means of
Test
–
SUT
Interaction
API
UI
Way of Captur ing Tests
+ No special skills
+ API not required
- Mostly complex, flaky,
expensive tools
+ Robust
+ More maintainable
+ Can be prebuilt
-
-
+ Simple, cheap tools
+ Somewhat robust
+ Fewer skills required
- API required
-
-
- More skills required
+ Mostly open-
source tools
Scripted
UI Tests
Built-in
R&PB
Modern
xUnit
Robot
User
Recorded Tests
Scripted Tests
- Very fragile
- Not maintainable
- Cannot be pre-built
- More skills required
- API required
- Less maintainable?
- Cannot be prebuilt
- Few COTS tools
- Somewhat fragile
- High maintenance
+ Can be prebuilt
+ UI is the
API
Scripted
UI Tests
Built -in
R&PB
Modern
XUnit
Robot
User
Means of
Test
–
SUT
Interaction
API
UI
Way of Captur ing Tests
+ No special skills
+ API not required
- Mostly complex, flaky,
expensive tools
+ Robust
+ More maintainable
+ Can be prebuilt
-
-
+ Simple, cheap tools
+ Somewhat robust
+ Fewer skills required
- API required
-
-
- More skills required
+ Mostly open-
source tools
Scripted
UI Tests
Built-in
R&PB
Modern
xUnit
Robot
User
Recorded Tests
Scripted Tests
- Very fragile
- Not maintainable
- Cannot be pre-built
- More skills required
- API required
- Less maintainable?
- Cannot be prebuilt
- Few COTS tools
- Somewhat fragile
- High maintenance
+ Can be prebuilt
+ UI is the
API
Scripted
UI Tests
Which Tools Do We Use to Automate Which Tests?
56
Chapter 6 Test Automation Strategy
could be applied to the UI components or units if the rest of the system can be
stubbed out.
Upper-Left Quadrant: Internal Recording
For completeness, the upper-left quadrant involves creating Recorded Tests via
an API somewhere behind the UI by recording all inputs and responses as the
SUT is exercised. It may even involve inserting observation points between the
SUT (at whatever granularity we are testing) and any DOCs. During test play-
back, the test APIs inject the inputs recorded earlier and compare the results
with what was recorded
This quadrant is not well populated with commercial tools
1
but is a feasible
option when building a Recorded Test mechanism into the application itself.
Introducing xUnit
The xUnit family of Test Automation Frameworks is designed for use in auto-
mating programmer tests. Its design is intended to meet the following goals:
• Make it easy for developers to write tests without needing to learn a
new programming language. xUnit is available in most languages in
use today.
• Make it easy to test individual classes and objects without needing to
have the rest of the application available. xUnit is designed to allow us
to test the software from the inside; we just have to design for testability
to take advantage of this capability.
• Make it easy to run one test or many tests with a single action. xUnit
includes the concept of a test suite and Suite of Suites (see Test Suite
Object on page 387) to support this kind of test execution.
• Minimize the cost of running the tests so programmers aren’t discour-
aged from running the existing tests. For this reason, each test should
be a Self-Checking Test (page 26) that implements the Hollywood
principle.
2
1
Most of the tools in this quadrant focus on recording regression tests by inserting obser-
vation points into a component-based application and recording the (remote) method calls
and responses between the components. This approach is becoming more popular with
the advent of service-oriented architecture (SOA).
2
The name is derived from what directors in Hollywood tell aspiring applicants at mass
casting calls: “Don’t call us; we’ll call you (if we want you).”
57
The xUnit family has been extraordinarily successful at meeting its goals.
I cannot imagine that Erich Gamma and Kent Beck could have possibly antici-
pated just how big an impact that fi rst version of JUnit would have on software
development!
3
The same characteristics that make xUnit particularly well suited
to automating programmer tests, however, may make it less suitable for writing
some other kinds of tests. In particular, the “stop on fi rst failure” behavior of as-
sertions in xUnit has often been criticized (or overridden) by people who want to
use xUnit for automating multistep customer tests so that they can see the whole
score (what worked and what didn’t) rather than merely the fi rst deviation from
the expected results. This disagreement points out several things:
• “Stop on fi rst failure” is a tool philosophy, not a characteristic of
unit tests. It so happens that most test automaters prefer to have their
unit tests stop on fi rst failure, and most recognize that customer tests
must necessarily be longer than unit tests.
•Itis possible to change the fundamental behavior of xUnit to satisfy
specifi c needs; this fl exibility is just one advantage of open-source
tools.
• Seeing a need to change the fundamental behavior of xUnit should
probably be interpreted as a trigger for considering whether some other
tool might possibly be a better fi t.
For example, the Fit framework has been designed specifi cally for the purpose
of running customer tests. It overcomes the limitations of xUnit that lead to the
“stop on fi rst failure” behavior by communicating the pass/fail status of each
step of a test using color coding. Another option for Java developers is TestNG,
which provides capabilities for explicitly sequencing Chained Tests (page 454).
Having said this, choosing a different tool doesn’t eliminate the need to make
many of the strategic decisions unless the tool constrains that decision making
in some way. For example, we still need to set up the test fi xture for a Fit test.
Some patterns—such as Chained Tests, where one test sets up the fi xture for a
subsequent test—are diffi cult to automate and may therefore be less attractive in
Fit than in xUnit. And isn’t it ironic that the very fl exibility of xUnit is what al-
lows test automaters to get themselves into so much trouble by creating Obscure
Tests (page 186) that result in High Test Maintenance Cost (page 265)?
3
Technically, SUnit came fi rst but it took JUnit and the “Test Infected” article [TI] to
really get things rolling.
Which Tools Do We Use to Automate Which Tests?