Meszaros G. xUnit Test Patterns Refactoring Test Code

Подождите немного. Документ загружается.

Data-Driven Test

How do we prepare automated tests for our software?

How do we reduce Test Code Duplication?

We store all the information needed for each test in a data ﬁ le and write an

interpreter that reads the ﬁ le and executes the tests.

Testing can be very repetitious not only because we must run the same test

over and over again, but also because many of the tests differ only slightly. For

example, we might want to run essentially the same test with slightly different

system inputs and verify that the actual output varies accordingly. Each of these

tests would consist of exactly the same steps. While having so many tests is an

excellent way to ensure good coverage of functionality, it is not so good for test

maintainability because any change made to the algorithm of one of these tests

must be propagated to all of the similar tests.

A Data-Driven Test is one way to get excellent coverage while minimizing

the amount of test code we need to write and maintain.

How It Works

We write a Data-Driven Test interpreter that contains all the common logic

from the tests. We put the data that varies from test to test into the Data-Driven

Test ﬁ le that the interpreter reads to execute the tests. For each test it performs

the same sequence of actions to implement the Four-Phase Test (page 358). First,

data

Fixture

Setup

Exercise

Ver ify

Teardown

SUT

Test 1

Data

Test 2

Data

Test n

Data

data

Fixture

Setup

Exercise

Ver ify

Teardown

SUT

Test 1

Data

Test 2

Data

Test n

Data

Data-Driven

Test

Chapter 18 Test Strategy Patterns

288

the interpreter retrieves the test data from the ﬁ le and sets up the test ﬁ xture us-

ing the data from the ﬁ le. Second, it exercises the SUT with whatever arguments

the ﬁ le speciﬁ es. Third, it compares the actual results produced by the SUT (e.g.,

returned values, post-test state) with the expected results from the ﬁ le. If the

results don’t match, it marks the test as failed; if the SUT throws an exception,

it catches the exception and marks the test accordingly and continues. Fourth,

the interpreter does any ﬁ xture teardown that is necessary and then moves on to

the next test in the ﬁ le.

A test that might otherwise require a series of complex steps can be reduced

to a single line of data in the Data-Driven Test ﬁ le. Fit is a popular example of

a framework for writing Data-Driven Tests.

When to Use It

A Data-Driven Test is an alternative strategy to a Recorded Test (page 278) and

a Scripted Test (page 285). It can also be used as part of a Scripted Test strategy,

however, and Recorded Tests are, in fact, Data-Driven Tests when they are played

back. A Data-Driven Test is an ideal strategy for getting business people involved

in writing automated tests. By keeping the format of the data ﬁ le simple, we make

it possible for the business person to populate the ﬁ le with data and execute the

tests without having to ask a technical person to write test code for each test.

We can consider using a Data-Driven Test as part of a Scripted Test strategy

whenever we have a lot of different data values with which we wish to exercise

the SUT where the same sequence of steps must be executed for each data value.

Usually, we discover this similarity over time and refactor ﬁ rst to a Parameterized

Test (page 607) and then to a Data-Driven Test. We may also want to arrange a

standard set of steps in different sequences with different data values much like

in an Incremental Tabular Test (see Parameterized Test). This approach gives us

the best coverage with the least amount of test code to maintain and makes it

very easy to add more tests as they are needed.

Another consideration when deciding whether to use Data-Driven Tests is

whether the behavior we are testing is hard-coded or driven by conﬁ guration

data. If we automate tests for data-driven behavior using Scripted Tests, we must

update the test programs whenever the conﬁ guration data changes. This behavior

is just plain unnatural because it implies that we must commit changes to our

source code repository [SCM] whenever we change the data in our conﬁ guration

database.

By making the tests data-driven, changes to the conﬁ guration data or

Of course, we should be managing our test data in a version-controlled Repository,

too—but that topic could ﬁ ll another book; see [RDb] for details.

Data-Driven Test

Data-Driven

Test

289

meta-objects are then driven by changes to the Data-Driven Tests—a much more

natural relationship.

Implementation Notes

Our implementation options depend on whether we are using a Data-Driven

Test as a distinct test strategy or as part of an xUnit-based strategy. Using a

Data-Driven Test as a stand-alone test strategy typically involves using open-

source tools such as Fit or commercial Recorded Test tools such as QTP. Using

a Data-Driven Test as part of a Scripted Test strategy may involve implementing a

Data-Driven Test interpreter within xUnit.

Regardless of which strategy we elect to follow, we should use the appropri-

ate Test Automation Framework (page 298) if one is available. By doing so, we

effectively convert our tests into two parts: the Data-Driven Test interpreter

and the Data-Driven Test ﬁ les. Both of these assets should be kept under ver-

sion control so that we can see how they have evolved over time and to allow

us to back out any misguided changes. It is particularly important to store the

Data-Driven Test ﬁ les in some kind of Repository, even though this concept may

be foreign to business users. We can make this operation transparent by provid-

ing the users with a Data-Driven Test ﬁ le-authoring tool such as FitNesse, or we

can set up a “user-friendly” Repository such as a document management system

that just happens to support version control as well.

It is also important to run these tests as part of the continuous integration

process to conﬁ rm that tests that once passed do not suddenly begin to fail.

Failing to do so can result in bugs creeping into the software undetected and

signiﬁ cant troubleshooting effort once the bugs are detected. Including the cus-

tomer tests in the continuous integration process requires some way to keep

track of which customer tests were already passing, because we don’t insist that

all customer tests pass before any code is committed. One option is to keep

two sets of input ﬁ les, migrating tests that pass from the “still red” ﬁ le into the

“all green” ﬁ le that is used for regression testing as part of the automatic build

process.

Variation: Data-Driven Test Framework (Fit)

We should consider using a prebuilt Data-Driven Test framework when we are

using Data-Driven Tests as a test strategy. Fit is a framework originally conceived

by Ward Cunningham as a way of involving business users in the automation

of tests. Although Fit is typically used to automate customer tests, it can also

be used for unit tests if the number of tests warrants building the necessary ﬁ x-

tures. Fit consists of two parts: the framework and a user-created ﬁ xture. The Fit

Data-Driven

Test

Chapter 18 Test Strategy Patterns

290

291

Framework is a generic Data-Driven Test interpreter that reads the input ﬁ le and

ﬁ nds all tables in it. It looks in the top-left cell of each table for a ﬁ xture classname

and then searches our test executable for that class. When it ﬁ nds a class, it creates

an instance of the class and passes control to that instance as it reads each row and

column of the table. We can override methods deﬁ ned by the framework to specify

what should happen for each cell in the table. A Fit ﬁ xture, then, is an adapter that

Fit calls to interpret a table of data and invoke methods on the SUT.

The Fit table can also contain expected results from the SUT. Fit compares

the speciﬁ ed values with the actual values returned by the SUT. Unlike Asser-

tion Methods (page 362) in xUnit, however, Fit does not abandon a test at the

ﬁ rst value that does not match the expected value. Instead, it colors in each cell

in the table, with green cells indicating actual values that matched the expected

values and red cells indicating wrong or unexpected values.

Using Fit offers several advantages:

• There is much less code to write than when we build our own test

Interpreter [GOF].

• The output makes sense to a business person, not just a technical person.

• The tests don’t stop at the ﬁ rst failed assertion. Fit has a way of com-

municating multiple failures/errors in a way that allows us to see the

failure patterns very easily.

• There are a plethora of ﬁ xture types available to subclass or use as is.

So why wouldn’t we use Fit for all our unit testing instead of xUnit? The main

disadvantages of using Fit are described here:

• The test scenarios need to be very well understood before we can build

the Fit ﬁ xture. We then need to translate each test’s logic into a tabular

representation; this isn’t always a good ﬁ t, especially for developers who

are used to thinking procedurally. While it may be appropriate to have

testers who can write the Fit ﬁ xtures for customer tests, this approach

wouldn’t be appropriate for true unit tests unless we had close to a 1:1

tester-to-developer ratio.

• The tests need to employ the same SUT interaction logic in each test.

run several different styles of tests, we would probably have to build one

or more different ﬁ xtures for each style of test. Building a new ﬁ xture

is typically more complex than writing a few Test Methods (page 348).

The tabular data must be injected into the SUT during the ﬁ xture setup or exercise SUT

phases or retrieved from the SUT during the result veriﬁ cation phase.

Data-Driven Test

Data-Driven

Test

Although many different ﬁ xture types are available to subclass or use

as is, their use in this way is yet another thing that developers would be

required to learn to do their jobs. Even then, not all unit tests are ame-

nable to automation using Fit.

• Fit tests aren’t normally integrated into developers’ regression tests that

are run via xUnit. Instead, these tests must be run separately—which

introduces the possibility that they will not be run at each check-in.

Some teams include Fit tests as part of their continuous integration

build process to partially mitigate this issue. Other teams have reported

great success having a second “customer” build service or server that

runs all the customer tests.

Each of these issues is potentially surmountable, of course. In general, xUnit is

a more appropriate framework for unit testing than Fit; the reverse is true for

customer tests.

Variation: Naive xUnit Test Interpreter

When we have a small number of Data-Driven Tests that we wish to run as part

of an xUnit-based Scripted Test strategy, the simplest implementation is to write

a Test Method containing a loop that reads one set of input data values from the

ﬁ le along with the expected results. This is the equivalent of converting a single

Parameterized Test and all its callers into a Tabular Test (see Parameterized

Test). As with a Tabular Test, this approach to building the Data-Driven Test

interpreter will result in a single Testcase Object (page 382) with many asser-

tions. This has several ramiﬁ cations:

• The entire set of Data-Driven Tests will count as a single test. Hence,

converting a set of Parameterized Tests into a single Data-Driven Test

will reduce the count of tests executed.

• We will stop executing the Data-Driven Test on the ﬁ rst failure or

error. As a consequence, we will lose a lot of our Defect Localization

(see page 22). Some variants of xUnit do allow us to specify that failed

assertions shouldn’t abort execution of the Test Method.

• We need to make sure our assertion failures tell us which subtest we

were executing when the failure occurred.

We could address the last two issues by including a try/catch statement inside the

loop but surrounding the test logic and then continuing the code’s execution.

Nevertheless, we still need to ﬁ nd a way to report the test results in a meaningful

way (e.g., “Failed subtests 1, 3, and 6 with . . .”).

Data-Driven

Test

Chapter 18 Test Strategy Patterns

292

To make it easier to extend the Data-Driven Test interpreter to handle sev-

eral different kinds of tests in the same data ﬁ le, we can include a “verb” or

“action word” as part of each entry in the data ﬁ le. The interpreter can then

dispatch to a different Parameterized Test based on the action word.

Variation: Test Suite Object Generator

We can avoid the “stop on ﬁ rst failure” problem associated with a Naive xUnit

Test Interpreter by having the suite method on the Test Suite Factory (see Test

Enumeration on page 399) fabricate the same Test Suite Object (page 387)

structure as the built-in mechanism for Test Discovery (page 393). To do so,

we build a Testcase Object for each entry in the Data-Driven Test ﬁ le and ini-

tialize each object with the test data for the particular test.

That object knows

how to execute the Parameterized Test with the data loaded into it when the

test suite was built. This ensures that the Data-Driven Test continues execut-

ing even after the ﬁ rst Testcase Object encounters an assertion failure. We can

then let the Test Runner (page 377) count the tests, errors, and failures in the

normal way.

Variation: Test Suite Object Simulator

An alternative to building the Test Suite Object is to create a Testcase Object

that behaves like one. This object reads the Data-Driven Test ﬁ le and iterates

over all the tests when asked to run. It must catch any exceptions thrown by

the Parameterized Test and continue executing the subsequent tests. When

ﬁ nished, the Testcase Object must report the correct number of tests, failures,

and errors back to the Test Runner. It also needs to implement any other meth-

ods on the standard test interface on which the Test Runner depends, such as

returning the number of tests in the “suite,” returning the name and status of

each test in the suite (for the Graphical Test Tree Explorer, see Test Runner),

and so forth.

Motivating Example

Let’s assume we have a set of tests as follows:

def test_extref

sourceXml = "<extref id='abc' />"

expectedHtml = "<a href='abc.html'>abc</a>"

generateAndVerifyHtml(sourceXml,expectedHtml,"<extref>")

This is very similar to how xUnit’s built-in Test Method Discovery (see Test Discovery)

mechanism works, except that we are passing in the test data in addition to the Test

Method name.

Data-Driven Test

Data-Driven

Test

293

end

def test_testterm_normal

sourceXml = "<testterm id='abc'/>"

expectedHtml = "<a href='abc.html'>abc</a>"

generateAndVerifyHtml(sourceXml,expectedHtml,"<testterm>")

end

def test_testterm_plural

sourceXml = "<testterms id='abc'/>"

expectedHtml = "<a href='abc.html'>abcs</a>"

generateAndVerifyHtml(sourceXml,expectedHtml,"<plural>")

end

The succinctness of these tests is made possible by deﬁ ning the Parameterized

Test as follows:

def generateAndVerifyHtml( sourceXml, expectedHtml,

message, &block)

mockFile = MockFile.new

sourceXml.delete!("\t")

@handler = setupHandler(sourceXml, mockFile )

block.call unless block == nil

@handler.printBodyContents

actual_html = mockFile.output

assert_equal_html( expectedHtml,

actual_html,

message + "html output")

actual_html

end

The main problem with these tests is that they are still written in code when, in

fact, the only difference between them is the data used as input.

Refactoring Notes

The solution, of course, is to extract the common logic of the Parameterized

Tests into a Data-Driven Test interpreter and to collect all sets of parameters

into a single data ﬁ le that can be edited by anyone. We need to write a “main”

test that knows which ﬁ le to read the test data from and a bit of logic to read

and parse the test ﬁ le. This logic can call our existing Parameterized Test logic

and let xUnit keep track of the test execution statistics for us.

Example: xUnit Data-Driven Test with XML Data File

In this example, we will use XML as our ﬁ le representation. Each test consists of

a test element with three main parts:

Data-Driven

Test

294

Chapter 18 Test Strategy Patterns

• An action that tells the Data-Driven Test interpreter which test logic to

run (e.g., crossref)

• The input to be passed to the SUT—in this case, the sourceXml element

• The HTML we expect the SUT to produce (in the expectedHtml element)

These three components are wrapped up in a testsuite element.

<action>crossref</action>

</sourceXml>

</expectedHtml>

</test>

<action>crossref</action>

</sourceXml>

</expectedHtml>

</test>

<action>crossref</action>

</sourceXml>

</expectedHtml>

</test>

</testsuite>

This XML ﬁ le could be edited by anyone with an XML editor without any concern

for introducing test logic errors. All the logic for verifying the expected outcome

is encapsulated by the Data-Driven Test interpreter in much the same way as it

would be by a Parameterized Test. For viewing purposes we could hide the XML

structure from the user by deﬁ ning a style sheet. In addition, many XML editors

will turn the XML into a form-based input to simplify editing.

To avoid dealing with the complexities of manipulating XML, the interpreter

can also use a CSV ﬁ le as input.

Data-Driven Test

Data-Driven

Test

295

Example: xUnit Data-Driven Test with CSV Input File

The test in the previous example would look like this as a CSV ﬁ le:

ID, Action, SourceXml, ExpectedHtml

Extref,crossref,<extref id='abc'/>,<a href='abc.html'>abc</a>

TTerm,crossref,<testterm id='abc'/>,<a href='abc.html'>abc</a>

TTerms,crossref,<testterms id='abc'/>,<a href='abc.html'>abcs</a>

The interpreter is relatively simple and is built on the logic we had already devel-

oped for our Parameterized Test. This version reads the CSV ﬁ le and uses Ruby’s

split function to parse each line.

def test_crossref

executeDataDrivenTest "CrossrefHandlerTest.txt"

end

def executeDataDrivenTest ﬁlename

dataFile = File.open(ﬁlename)

dataFile.each_line do | line |

desc, action, part2 = line.split(",")

sourceXml, expectedHtml, leftOver = part2.split(",")

if "crossref"==action.strip

generateAndVerifyHtml sourceXml, expectedHtml, desc

else # new "verbs" go before here as elsif's

report_error( "unknown action" + action.strip )

end

Unless we changed the implementation of generateAndVerifyHtml to catch assertion

failures and increment a failure counter, this Data-Driven Test will stop executing

at the ﬁ rst failed assertion. While this behavior would be acceptable for regres-

sion testing, it would not provide very good Defect Localization.

Example: Data-Driven Test Using Fit Framework

If we wanted to have even more control over what the user can do, we could

create a Fit “column ﬁ xture” with the columns “id,” “action,” “source XML,”

and “expected Html()” and let the user edit an HTML Web page instead

(Figure 18.1).

Data-Driven

Test

Chapter 18 Test Strategy Patterns

296

[Figure 18.1: ‘CrossrefHandlerFitTest.vsd’]

Figure 18.1 A Data-Driven test built using the Fit framework.

When using Fit, the test interpreter is the Fit framework extended by the Fit

ﬁ xture class speciﬁ c to the test:

public class CrossrefHandlerFixture extends ColumnFixture {

// Input columns

public String id;

public String action;

public String sourceXML;

// Output columns

public String expectedHtml() {

return generateHtml(sourceXML);

}

The methods of this ﬁ xture class are called by the Fit framework for each cell

in each line in the Fit table based on the column headers. Simple names are

interpreted as the instance variable of the ﬁ xture (e.g., “id,” “source XML”).

Column names ending in “()” signify a function that Fit calls and then compares

its result with the contents of the cell.

The resulting output is shown in Figure 18.2. This colored-in table allows us

to get an overview of the results of running one ﬁ le of tests at a single glance.

Figure 18.2 The results of executing the Fit test.

Data-Driven Test

Data-Driven

Test

297