Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

Подождите немного. Документ загружается.

SYSTEM: What else can I help you with? [

pause

]

SER: I need to get the balance on another account.

YSTEM: I’m sorry. Please choose one of the following ...

In the first prompt, the open-ended question is intended simply as a friendly

and natural-sounding introduction to the list of choices that would have followed

the pause. But the user assumes that he should just answer the question being

asked and barges in before hearing the choices. This is not a case of the user incor-

rectly having high expectations for a system. On the contrary, the user’s expecta-

tions have rightly been built up through a series of positive interactions. The

problem here is that asking the open-ended question leads the user to believe that

the system will be able to understand his response.

The type of interaction shown here can be particularly damaging because it

erodes the trust and confidence the user has built up with the system. Asking a

question and then not being able to deal with the answer makes users feel as if

they have been deliberately tricked into doing something foolish.

Once an error like this occurs, it can be extremely difficult to get the interac-

tion flowing smoothly again. Once users become aware of the previously hidden

limitations of the system, they become attentive for other instances in which

the system shows its lack of conversational ability. As the user builds up positive

interactions with a system, the conversation seems more and more realistic and

the system seems more human. But before the user can reach the uncanny valley

(in which almost-human computer systems go from arousing positive emotions to

negative ones [Mori, 1970]), the illusion often shatters due to the types of issues

described in this section. The end result is too often what has been called the “con-

versational death spiral” (Attwater et al., 2005) in which the system and the user

both try unsuccessfully to repair the conversation via disparate strategies.

6.1.3 The Presentation Layer: Prompts

Users experience the VUI primarily through its prompts. Prompts are what the

VUI says and are the entirety of the evidence users receive about their interlocu-

tor. Through prompts, users experience the voice of the system, as well as linguis-

tic and prosodic features that contribute to the attribution of a persona. The best

way to appreciate the impact of prompts is to experience them in an automated

conversation. Imagine the following situation. A user, Nick, is having trouble with

his satellite television service—his television screen goes blank intermittently on

There are speech recognition systems designed to handle responses to open-ended ques-

tions. The system described here is typical of grammar-based recognition systems that rec-

ognize only items preloaded into the grammar.

6 Voice User Interfaces

202

several channels. Nick goes online and reads the troubleshooting information

available there, but does not find anything useful. He then decides to call his

satellite provider at the number helpfully provided on the web page. Table 6.1

contains an account of his phone call.

Nick’s call is typical of the interactions that many users have with VUIs.

Users begin the conversation cooperatively, and when something goes wrong they

turn to conversation repair techniques that work in other conversations (e.g.,

speaking more slowly and clearly as in Step 12 in the table). Ironically these repair

TABLE 6.1 Hypothetical Call to a Satellite Television Provider

Step Speaker Utterance

User’s Thoughts and Actions/

Comments

1 System Hello and thank you for calling Best

Choice Satellite

(branding sound)

, the

best choice for all your satellite

television needs.

So far, so good.

2 System You can also contact us via our website,

w-w-w dot best choice dot com. Our

website offers complete information

about setting up satellite service at

your home and offers valuable

installation coupons worth over $40.

This is really annoying! I just spent 10

minutes on the website without solving

my problem.

3 System I can help you with the following five

options. You can interrupt me and

speak your choice at any time. Please

select one of the following: Sign up for

new service, add features to my

service, move my existing service,

problems with my satellite service, or

ask a billing question.

I’m glad the guy behind the counter at the

local satellite TV office doesn’t talk this

much.... Oh wait, which one do I

want?

4 Nick Service problem.

Finally!

5 System I’m sorry. I did not understand your

response. I can help you with the

following five options. You can interrupt

me and speak your choice at any time.

Please select one of the following: Sign

up for new service, add features to my

service, move my existing service,

problems with my satellite service, or

ask a billing question.

Huh? Better listen again to figure out

exactly what to say.

6 Nick Problems with my satellite service.

(

Continued

)

6.1 Automated Conversation: Human versus Machine

203

techniques often exacerbate the original problem or create new ones. In the fol-

lowing, the conversation is analyzed turn by turn to highlight what went wrong

and how to avoid the problem. The design guidelines in Section 6.6 detail many

of the principles introduced here.

Step 1 of the conversation actually begins with a useful practice. The recorded

greeting incorporates a nonspeech branding sound (e.g., the Intel noise). This serves

the useful function of demonstrating that the user is interacting with an automated

system, not a real person. Without the nonspeech audio, the user may be initially

unsure whether he is talking to a live agent faithfully reciting the official company

greeting or listening to a recording. Some systems choose to announce their nonhu-

man status (e.g., “You have reached ABC Bank’s 24-hour automated account access

TABLE 6.1

Cont’d

Step Speaker Utterance

User’s Thoughts and Actions/

Comments

7 System Please say or enter your account number.

I guess she heard me. But I don’t know

my account number! Where am I going

to find it?

8 Nick

(mutters)

Just a second ...

(Nick frantically shuffles papers trying to

find his last satellite TV bill and mutters

“Just a second” under his breath.)

9 System I’m sorry. The number you entered is not

a valid account number. Please say or

enter your account number.

Why did she think that was an account

number? Anyone can tell that “Just a

second” is not an account number!

Whatever, I’ll just try again.

10 Nick 0-0-0-0-dash-4-3-5-7-2-2-7-6

(Nick reads what he believes to be his

account number from his printed

billing statement.)

11 System I’m sorry. The number you entered is not

a valid account number. Please say or

enter your account number.

(System fails to recognize because

recognized account numbers are eight

digits and do not include leading zeros

and the hyphen. System goes to a

reprompt.)

Give me a break. I just told you the

account number.

12 Nick 0... 0... 0... 4... 3... 5... 7... 2...

This machine is idiotic. I’m going to go

extra slow so she can’t mess up again.

13 System I’m sorry, that is not a valid account

number. One moment while I transfer

your call.

6 Voice User Interfaces

204

system”). Although a greeting like this can serve the same informative purpose, non-

speech audio in the greeting avoids immediately arousing the negative emotions

some users experience when they hear they will be talking to a system.

Step 2 demonstrates the common practice of using the moment just following

the greeting to insert advertisements or give instructions on how to use the system.

The idea here is that users should be given this information early, before they start

using the system, just in case they need it later. However, information presented to

users out of context will not seem relevant and is unlikely to be retained. When users

begin an interaction with an automated system, they tend to be extremely task

focused—they are thinking about what they need to accomplish and nothing else.

My experience from usability testing suggests that users ignore information

presented before they locate the functionality they need to accomplish their task.

In the case of advertising messages, it is clear that the placement of the message

is ineffective because the audience is primed to ignore it until they reach their

goal. This reasoning tends to be persuasive when negotiating with marketing

departments—simply ask if they intend their ad to actively annoy the user.

The fictional menu shown in Step 3 is based on the menus one often hears in

VUIs:

I can help you with the following five options. You can interrupt me and

speak your choice at any time. Please select one of the following: sign up

for new service, add features to my service, move my existing service,

problems with my satellite service, or ask a billing question.

There are only five menu options, but the menu still seems quite long. (See

Section 6.6.1 for a discussion of menus and memory.) One reason is that there

are three sentences of preamble before the options are actually presented.

As with advertisements discussed in the previous section, instructions pre-

sented before the main menu are typically ignored by users. These introductory

prompts are often thought to be necessary to prepare users who may be using a

speech system for the first time and who are more accustomed to touch-tone sys-

tems. VUI best practices suggest that most users will be able to give satisfactory

spoken responses as long as the prompts include wording like “You can say ...”

before menu options are presented. Another problem with the menu presented

in Step 3 is the menu option names themselves. Each option is long and syntacti-

cally complex, and the option names are partially overlapping (four use the term

“service”). This makes it difficult for users to locate and remember the menu

option they want.

The user’s response at Step 4—“Service problem”—demonstrates the sort of

responses generated by long, complex menu option names. The user accurately

identified the option he wanted, and produced an abbreviated version of it as

his response, which was unfortunately not recognized by the system. Because

the user accurately selected the appropriate option, the menu option in itself is

6.1 Automated Conversation: Human versus Machine

205

not fully at fault. Instead, this represents a failure in the grammar for this menu.

A grammar is the set of all items that can be recognized at a specific recognition

state, as defined by the VUI designer. It is easy for a person to understand that

the user’s response “Service problem” was intended as a synonym for the menu

option “Problems with my satellite service.” However, the speech system is unable

to make the same judgment unless that particular synonym is built into the gram-

mar. The longer and more complex the menu option names, the more possible

synonyms there are, and thus longer and more complex the grammars are

required to support the menu options.

Step 5 is an example of ineffective error handling:

I’m sorry. I did not understand your response. I can help you with the fol-

lowing five options. You can interrupt me and speak your choice at any

time. Please select one of the following: sign up for new service, add fea-

tures to my service, move my existing service, problems with my satellite

service, or ask a billing question.

This reprompt blames the user for the error by suggesting that he failed to cor-

rectly choose an item from the menu, when in fact the user is giving a predictable

variant of a menu option name that should have been covered in the grammar.

The reprompt then repeats the entire initial prompt, including the introductory

sentences, but does not offer any information to the user about specifically how

to give a well-formed response. Simple repetition is an effective reprompting strat-

egy only for those cases in which the user fails to respond because he did not hear

the choices initially. Here, the user clearly tried to make a response, so the

reprompt is simply annoying.

Why is the user feeling uncomfortable in Step 7?

Please say or enter your account number.

In part it is because he is being asked for information that he does not have at hand.

Even before this, though, the user is uncomfortable with the flow of the conversa-

tion because he is not sure whether his response in the previous step was accepted.

What’s missing is a seemingly unimportant connector word, acknowledging the

user’s response. Simply adding a word or phrase such as “All right” before asking

for the account number would have let the user know that he was heard and under-

stood, and that the conversation would be moving on to a new topic.

These connector words, known as

discourse markers,

serve many functions

in spoken conversation, including acknowledgment (okay, all right), changing

topics (otherwise, instead), emphasizing or reinforcing (also, above all), and

serving as sequence markers (first, next, finally) (Schiffrin, 1987). Discourse mar-

kers convey vital information about the structure and flow of the overall dialog.

Without discourse markers, a conversation feels incomplete, and will leave users

uncomfortable without knowing exactly why.

6 Voice User Interfaces

206

Step 8, in which the user mutters “Just a second,” is both an indication of the

ineffectiveness of the account number prompt in Step 7 and an illustration of the

blatant lack of real-world knowledge of speech recognition systems. To any per-

son, it is obvious that the user was not intending “Just a second” as a response

to the account number prompt. He was simply mumbling as he tried to locate

his account number. The system cannot at face value distinguish system-directed

speech from side speech (defined as speech audible over the telephone connec-

tion that is not intended as part of the telephone conversation). The fact that

the user is muttering here is a sign that there is a problem with the account num-

ber prompt. The user hesitates because he is unprepared to give the information

he is being asked to provide. The best way to prevent this sort of problem is to

avoid asking for information that the user will need to look up, in favor of some-

thing the user will have memorized such as a telephone number. When this is

impossible, give the user practical advice for where they can find and recognize

the requested information.

Step 9 is another poorly designed reprompt:

I’m sorry. The number you entered is not a valid account number. Please

say or enter your account number.

First, the user did not in fact enter an account number, so the prompt should not

accuse him of entering an invalid one. More importantly, the reprompt fails to

offer the user assistance in finding the account number. Especially when request-

ing specific pieces of information, it is vital to write reprompts that give users

additional instruction that will help them provide a well-formed response. The

user would have been more successful and more comfortable if the prompt had

said, for example, “Please say or enter your eight-digit account number. You can

find your account number in the blue box under your name on your statement.”

Step 10 shows the results of giving the user insufficient instruction about what

constitutes an account number. Nick located the number and said, “0-0-0-0-dash-4-

3-5-7-2-2-7-6” because he was not instructed to speak only the last eight digits. The

grammar contributed to the problem at this step because it did not accommodate

the full 12-digit version of the account number.

Step 11 is the identical reprompt played in Step 9. What was annoying and

unhelpful on first presentation can become infuriating with repetition. Repeating

the identical reprompt, with the identical cheerful tone of voice, is completely out

of sync with the conversation. The user has just tried multiple times to give a

piece of information and failed in each case. When the system calmly repeats

the identical prompt, the user can feel as if he is being patronized. When

writing reprompts, be mindful of the user’s potential frustration level. Two tries

at any given recognition state is about the maximum that users can tolerate with-

out significant annoyance, in my experience. Earlier I recommended brief

reprompting for a reprompt at a menu (see Section 6.1.2). Here, and for other

6.1 Automated Conversation: Human versus Machine

207

states when the user is inputting data (account or ID numbers, etc.), reprompts

should actually be more expansive than the initial prompt, adding information

to help the user provide a well-formed response.

In Step 12 when Nick slowly says, “0... 0... 0... 4... 3... 5... 7... 2...,” we

see the expectations mismatch in action. In frustration, the user falls back on a

strategy for giving information that works in human-to-human conversations

under these circumstances—speaking slowly and clearly. In the automated con-

versation, however, this strategy makes the recognition problem worse. Acoustic

models used in recognition are built on normal conversational speech, not on

the deliberate, slow, overenunciated speech spoken here; thus, recognition may

suffer. In addition, the user is speaking so slowly that he has exceeded the sys-

tem’s time limit for giving a response. Consequently, the system interrupts the

user before he is done speaking, adding a turn-taking error (Sachs, Schleghoff, &

Jefferson, 1974) to the recognition problems. The problem can be reduced by

adjusting the parameter that specifies the amount of time the user has to respond

before the system times out at a dialog state (sometimes labeled “max speech

timeout”). It is far preferable to avoid causing the user frustration in the first place

by limiting the number of retries and using more effective reprompts.

In Step 13 the user is released from speech technology jail and sent to wait in

a queue for a live representative after several minutes of torture and no closer to

solving his problem. The sort of frustration and poor customer service experi-

enced by the imaginary user here is the motivation for real users to advocate for

IVR reform (for both speech and touch-tone) through the “gethuman” initiative

(

http://www.gethuman.com

6.2

TECHNOLOGY OF THE INTERFACE

The technology behind VUIs is automatic speech recognition (ASR). The system is

often a speech-enabled IVR, but increasingly may also be a personal computer,

in-car navigation system, or other speech-enabled handheld device. All these sys-

tems depend on the same basic technology to recognize speech input. In each

case, the speech signal is captured, digitized, segmented, and then compared

against a set of stored acoustic models for speech sounds. The sounds are then

built up into potential words that are compared with a grammar of words that

are to be recognized at that point in the dialog. The recognition process is acous-

tically driven—there is no top-down semantic or logical analysis to help the sys-

tem decode user input. Thus, the way that computer systems recognize speech

is fundamentally different from the way that humans do. Most ASR systems in

commercial use today rely on Hidden Markov Models (Rabiner, 1989) to recognize

words.

In the 1980s and 1990s, a great deal of research was focused on building more

human-like speech recognitions systems (e.g., McClelland & Elman, 1986). These

6 Voice User Interfaces

208

systems aimed at mimicking the multiple sources of nonacoustic information used

by humans when decoding the acoustic speech signal. Such systems contained

huge amounts of linguistic information about the structure of words, sentence

structure, and semantics (meaning at the level of words and sentences). Even this

was insufficient to model everything that humans bring to the task of speech

recognition, so models of dialog structure and world knowledge were also

incorporated. The complexity and computational requirements of such systems

have generally kept them in the laboratory rather than allowing them to migrate

into commercial applications.

6.2.1 The Recognition Process

Figure 6.1 depicts the steps in the recognition process. User input is captured via a

microphone (either built into the phone or connected to the computer or other

device being used). The first major technical hurdle for speech systems is encoun-

tered at this very early stage of the process: What counts as user input? The pro-

cess of making this decision is called end-pointing. A microphone will capture

any sound in the environment, not just those deliberately uttered by the user to

the system. In noisy environments, the quality of the audio signal is poorer than

the signal captured in a quiet environment. We have all tried using mobile phones

in noisy environments (in a car in traffic, at a crowded airport) and experienced

problems related to background noise, even when speaking to another human.

For speech recognition systems, this problem is amplified because systems cannot

automatically distinguish speech from nonspeech. This issue can be improved by

simply making the threshold for recognition of any sound higher, so that low-level

background noises are not detected. There is a trade-off, however, in this method

because users who speak softly will simply not be heard by a system with too high

a threshold.

End-pointing

Speech

End-pointed

utterance

Word

strin

Meaning

Actions

prompts

Feature

vectors

Feature

extraction

Recognition

Natural-

language

understanding

“I wanna go

to Dallas”

Destination: Dallas

Play> OK, what

day do you want

to leave?

Dialog

management

FIGURE

6.1

Speech recognition process.

Sequence of steps in automatic speech recognition.

Source:

From Cohen,

Giangola, and Balogh (2004).

6.2 Technology of the Interface

209

In addition to background noise, other classes of sound complicate the cap-

ture process. Nonspeech “mouth sounds” such as coughing, clearing the throat,

and laughing are particularly hard to distinguish because they share some of the

acoustic qualities of speech in that they are produced by the human vocal tract.

The most disruptive to conversations with speech systems is side speech—speech

that is within range of the microphone but is not directed at the system. Side

speech may be spoken by the user or by another person in the room with the user.

Side speech is disruptive not only because is begins a round of error handling, but

also because it emphasizes the limitations of the system. When the system cannot

distinguish between speech that is logically part of the conversation and speech

that is meant to be ignored (e.g., Nick’s “Just a second” in Table 6.1), the user is

suddenly acutely aware of how “stupid” the system really is. Something users per-

ceive as completely effortless—knowing what is and is not part of the current con-

versation—is beyond the capabilities of the automated system, so users begin to

doubt the system’s abilities overall.

Once the signal is captured, it is digitized and divided into a set of short seg-

ments for acoustic analysis. Each segment is transformed into a feature vector—

a numerical representation of the speech signal that contains information relevant

to recognition. The sequence of feature vectors is the basis for recognition at the

level of words.

The elements that contribute to the recognition model are acoustic models,

dictionaries, grammars, and a search algorithm. Acoustic models specify the pro-

nunciation of phonemes—the speech sounds that comprise a language—in terms

of sequences of feature vectors. Acoustic models are generally built by training

them against large sets of transcribed and labeled acoustic data. Thus, acoustic

models only support pronunciations that are well represented in the training data.

This is the reason that unusual accents, or particular voices, are sometimes poorly

recognized by speech systems.

A dictionary in a speech recognition system is a list of words and associated

pronunciations. Pronunciations are represented using a phonetic alphabet, and there

may be several alternate pronunciations for a single word. Pronunciation of words

may vary depending on factors like dialect, rate of speech, and speaking style. For

a specific pronunciation to be recognized, it must be added to the dictionary.

A grammar is the list of utterances that the recognition system is expected to

receive as input at each recognition state. Every time we intend the recognizer

to accept incoming speech, we must define what it is that callers are likely to

say. A grammar should list all the likely variants of responses that users might

produce at a given state (recall Step 4 in Nick’s conversation, Table 6.1). There

is a cost to adding items to a grammar, however; the more items there are in a

grammar and the more similar the grammar items are acoustically, the more dif-

ficult recognition will be. Grammars that are written by a VUI designer or devel-

oper are called rule-based grammars, and are used in the majority of recognition

applications in commercial use.

6 Voice User Interfaces

210

A second sort of grammar is defined statistically rather than by rule. Statisti-

cal language models (SLMs) are created automatically from transcribed speech

data. The SLM looks at all the possible sequences of words and computes the prob-

ability of words appearing in particular contexts. SLMs allow users to give rela-

tively free-form responses rather than simply repeating menu options offered by

the system. This makes SLMs attractive to many organizations deploying speech

technology. (Remember that

Star Trek

model? Speech recognition software sales

folks can convince you that SLMs almost make it possible.) To be effective, how-

ever, SLMs require vast amounts of training data (tens or hundreds of thousands

of utterances), which must all be transcribed and labeled by a person. SLMs are

therefore several orders of magnitude more time consuming and costly than

rule-based grammars. Only large organizations tend to see a positive return on

investment from SLM-based speech recognition.

A final type of grammar to note is slotted grammars, in which the user can

speak several individual pieces of information in a single utterance. Slotted gram-

mars are often used for simple transactions such as transferring funds between

accounts or requesting a travel reservation between two cities. User utterances

for slotted grammars can seem quite free-form, similar to the possible responses

for statistical language models, such as “I want to transfer $3,000 from Market

Growth Fund to International Equity Fund.” However, slotted grammars are much

more limited because they are designed to collect multiple discrete chunks of

information that can be connected by filler words. In the simple transfer example

above, a slotted grammar would be built to search the incoming audio signal to fill

an amount slot, a from-fund slot, and then a to-fund slot. If a user response devi-

ates from this amount ! from-fund ! to-fund model, a slotted grammar will fail

to recognize it, showing the very limited range of input flexibility possible for slot-

ted grammars. This limitation makes slotted grammars a good fit for tasks where

user responses are formulaic and follow a set pattern (such as simple transfers),

but less suitable for situations in which users might phrase their response in many

different ways.

Using the acoustic models, dictionaries, and grammars, the speech recogni-

tion system then looks for matches between the sequence of feature vectors (that

represent incoming speech) and possible words and pronunciations in the gram-

mar. The best matching path through the sequence of feature vectors is the word

that is recognized. Most speech recognizers have the ability to return not just the

best matching path, but

-best paths (the

-best matching paths rather than just

the top of the list). Associated with each item in the

-best list will be a confi-

dence score, a measure of how confident the recognizer is that the match is cor-

rect. In practice, the

-best list is a rank-ordered list of words and the probability

for each. The method for searching for best paths used most often in current

speech recognition systems is called Hidden Markov Models (HMMs). HMMs

are a statistical technique for computing the probability of a match at a state

and between states.

6.2 Technology of the Interface

211