Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

Подождите немного. Документ загружается.

Hart, S. G., and Staveland, L. E. (1988). Development of NASA TLX (Task Load Index):

Results of empirical and theoretical research. In Hancock, P. A., & Meshkati, N., eds.,

Human Mental Workload

. Amsterdam: North-Holland, 139–83.

Hayward, C. (1994). Listening to the earth sing. In Kramer, G., ed.,

Auditory Display:

Sonification, Audification and Auditory Interfaces

. Reading, MA: Addison-Wesley,

369–404.

Hermann, T. (2002). Sonification for exploratory data analysis. PhD diss., Bielefeld

University.

Hermann, T., & Hunt, A. (2005). An introduction to interactive sonific ation.

IEEE Multi-

media

12(2):20–24.

Hermann, T., Bai er, G ., Step hani, U ., & Ritter, H. (2006). Vocal sonification of patho-

logic EEG features.

Proceedings of Internati onal Conference on Auditory Display

Lond on.

Hermann, T., Drees, J. M., & Ritter, H. (2005). Broadcasting auditory weather reports—a

pilot project.

Proceedings of International Conference on Auditory Display

, Boston.

Hix, D., & Hartson, H. R. (1993). Formative evaluation: Ensuring usability in user inter-

faces. In Bass, L., & Dewan, P., eds.,

User Interface Software

. Chichester, England; New

York: Wiley.

Hunt, A., & Hermann, T. (2004). The importance of interaction in sonification.

Proceedings

of International Conference on Auditory Display

, Sydney.

International Community for Auditory Display (ICAD). (2007). Home page—

http://www.

icad.org/

Janata, P., & Childs, E. (2004). Marketbuzz: Sonification of real-time financial data.

Proceed-

ings of International Conference on Auditory Display

, Sydney.

Jones, D. G., and Endsley, M. R. (2000). Overcoming representational errors in complex

environments.

Human Factors

42(3):367–78.

Kidd Jr., G., Mason, C. R., & Arbogast, T. L. (2002). Similarity, uncertainty, and masking in

the identification of nonspeech auditory patterns.

Journal of the Acoustical Society of

America

111(3):1367–76.

Kramer, G., ed. (1994).

Auditory Display: Sonification, Audification and Auditory Interfaces

Reading, MA: Addison-Wesley.

Leplatre, G., & McGregor, I. (2004). How to tackle auditory interface aesthetics? Discussion

and case study.

Proceedings of International Conference on Auditory Display

, Sydney.

Lieder, C. N. (2004).

Digital Audio Workstation

. New York: McGraw-Hill.

Lumsden, J., Brewster, S. A., Crease, M., & Gray, P. D. (2002). Guidelines for audio-

enhancement of graphical user interface widgets.

Proceedings of People and Computers

XVI—Memorable Yet Invisible: Human Computer Interaction

, London.

Mankoff, J., Dey, A. K., Hsieh, G., Kientz, J., Ames, M., & Lederer, S. (2003). Heuristic eval-

uation of ambient displays.

Proceedings of SIGCHI Conference on Human Factors in

Computing Systems

, Fort Lauderdale.

Martinson, E., & Brock, D. (2007). Improving human–robot interaction through adaptation

to the auditory scene.

Proceedings of International Conference on Human–Robot Interac-

tion

, Washington, DC.

Massof, R. W. (2003). Auditory assistive devices for the blind.

Proceedings of International

Conference on Auditory Display

, Boston.

5 Auditory Interfaces

192

McAdams, S., & Bigand, E., eds. (1993).

Thinking in Sound: The Cognitive Psychology of

Human Audition

. Oxford; New York: Oxford University Press.

McCartney, J. (1996). SuperCollider Hub—available at

http://supercollider.sourcefor ge.

net

McFarlane, D. C., & Latorella, K. A. (2002). The scope and importanc e of human interrup-

tion in HCI design.

Human–Computer Interaction

17(1):1–61.

McGookin, D. K., & Brewster, S. A. (2006a). Haptic and audio interaction design. First Inter-

national Workshop, HAID 2006, Glasgow.

Proceedings (Lecture Notes in Computer Sci-

ence)

. New York: Springer-Verlag.

McGookin, D. K., & Brewster, S. A. (2006b). Advantages and issues with concurrent audio

presentation as part of an auditory display.

Proceedings of International Conference on

Auditory Displays

, London.

Mills, A. W. (1958). On the minimum audible angle.

Journal of the Acoustical Society of

America

30(4):237–46.

Miranda, E. R. (1998).

Computer Sound Synthesis for the Electronic Musi cian

.Oxford:FocalPress.

Mitsopoulos, E. N. (2000). a principled approach to the design of auditory interaction in the

non-visual user interface. PhD diss., University of York.

Moore, B. C. J. (2003).

An Introduction to the Psychology of Hearing

, 5th ed. London: Aca-

demic Press.

Moore, F. R. (1990).

Elements of Computer Music

. Upper Saddle River, NJ: Prentice Hall.

Mossberg, W. (2006). New earphone devices let you go cordless on iPods, cellphones. Per-

sonal Technology.

The Wall Street Journal Online

, December 21; available at

http://

ptech.wsj.com/archive/ptech–20061221.html

Mynatt, E. (1995). Transforming graphical interfaces into auditory interfaces. PhD diss.,

Georgia Institute of Technology, Atlanta.

Neisser, U. (1967).

Cognitive Psychology

. New York: Appleton-Century-Crofts.

Neuhoff, J. (2004). Interacting auditory dimensions. In J. Neu hoff, ed.,

Ecological Psycho-

acoustics

. Boston: Elsevier/Academic Press.

Nielsen, J. (1994). Ten usability heuristics—available at

http://www.useit.com/papers/

heuristic/heuristic_list.htm

Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces.

Proceedings of

SIGCHI Conference on Human Factors in Computing Systems

Peres, S. C., & Lane, D. M. (2005). Auditory graphs: The effects of redundant dimensions

and divided attention.

Proceedings of International Conference on Auditory Displays,

Limerick, Ireland.

Petrie, H., & Morley, S. (1998). The use of non-speech sounds in non-visual interfaces to the

MS-Windows GUI for blind computer users.

Proceedings of International Conference on

Auditory Displays

, Glasgow.

Pirhonen, A., Murphy, E., McAllister, G., & Yu, W. (2006). Non-speech sounds as elements

of a use scenario: A semiotic perspective.

Proceedings of International Conference on

Auditory Displays

, London.

Plomp, R. (1964). Rate of decay of auditory sensation.

Journal of the Acoustical Society of

America

36:277–82.

Pollack, I., & Ficks, L. (1954). Information of elementary multidimensional auditory dis-

plays.

Journal of the Acoustical Society of America

26:155–58.

References

193

Puckette, M. S. (1997). Pure Data.

Proceedings of International Computer Music Confer-

ence

, 224–27.

Pulkki, V. (1997). Virtual sound source positioning using vector base amplitude panning.

Journal of the Audio Engineering Society

45(6):456–66.

Recarte, M. A., & Nunes, L. M. (2003). Mental workload while driving: Effects on visual

search, discrimination, and decision making.

Journal of Experimental Psychology,

Applied

9(2):119–37.

Roads, C. (2001).

Microsound

. Cambridge, MA: MIT Press.

Sawhney, N., & Schmandt, C. (2003). Nomadic Radio: Wearable Audio Computing.

Cambridge, MA: Speech Interface Group, MIT Media Laboratory; available at

http://

web.media.mit.edu/~nitin/NomadicRadio/

Schmandt, C. (1993). Phoneshell: The telephone as computer terminal.

Proceedings of Asso-

ciation for Computing Machinery International Conference on Multimedia

Shannon, R. V., Zeng, F., Kamath, V., Wygonski, J., & Ekelid, M. (1995). Speech recognition

with primarily temporal cues.

Science

270(5234):303–304.

Staal, M. A. (2004). Stress, cognition, and human performance: A literature review and con-

ceptual framework. National Aeronautics and Space Administration (NASA) Technical

Memorandum, TM-2004-212824. Moffett Field, CA: Ames Research Center.

Stanton, N. A., & Edworthy, J., eds. (1999).

Human Factors in Auditory Warnings

. Alder-

shot, Hampshire, England: Ashgate Publishers.

Stevens, S. S. (1957). On the psychophysical law.

Psychological Review

64(3):153–81.

tenHave, P. (1999).

Doing Conversation Analysis: A Practical Guide

. London: Sage.

Terenzi, F. (1988). Design and realization of an integrated system for the composition of

musical scores and for the numerical synthesis of sound (special application for transla-

tion of radiation from galaxies into sound using computer music procedures). Physics

Department, University of Milan, Milan, Italy.

Truax, B., ed. (1999).

Handbook for Acoustic Ecology

; see

http://www2.sfu.ca/sonic-studio/

handbook/

Ulfvengren, P. (2003). Design of natural warning sounds in human machine systems. PhD

diss., Royal Institute of Technology, Stockholm.

VRSonic. (2007). VibeStudio—availabl e at

http://www.vrs onic.com

Walker, B. N. (2002). Magnitude estimation of conceptual data dimensions for use in sonifi-

cation.

Journal of Experimental Psychology: Applied

8(4):211–21.

Walker, B. N., & Kramer, G. (2004). Ecological psychoacoustics and auditory displays:

Hearing, grouping, and meaning making. In Neuhoff, J., ed.,

Ecological Psychoacoustics

New York: Academic Press, 150–75.

Walker, B. N., & Lane, D. M. (2001). Psychophysical scaling of sonification mappings:

A comparison of visually impaired and sighted listeners.

Proceedings of International

Conference on Auditory Display

, Espoo, Finland.

Walker, B. N., & Lindsay, J. (2004). Auditory navigation performance is affected by way-

point capture radius.

Proceedings of International Conference on Auditory Display

Sydney.

Walker, B. N., & Stamper, K. (2005). Building audio designs monkey: An audio augmented

reality designer’s tool.

Proceedings of International Conference on Auditory Display

Limerick, Ireland.

5 Auditory Interfaces

194

Walker, B. N., Nancy, A., & Lindsay, J. (2006). Spearcons: Speech-based earcons improve

navigation performance in auditory menus.

Proceedings of International Conference on

Auditory Display

, London.

Wang, D., & Brown, G. J. (2006).

Computational Auditory Scene Analysis: Principles, Algo-

rithms, and Applications

. Hoboken, NJ: John Wiley & Sons.

Weir, J. (2005). The sound of traffic—avai lable at

http://www.smokinggun.com/projects/

soundoftraffic/

Wickens, C. D., & Hollands, J. G. (2000).

Engineering Psychology and Human Performance

3rd ed. Upper Saddle River, NJ: Prentice Hall.

Williamson, J., Murray-Smith, R., & Hughes, S. (2007). Shoogle: Multimodal excitatory inter-

faces on mobile devices.

Proceedings of SIGCHI Conference on Human Factors in Com-

puting Systems

Winberg, F., & Bowers, J. (2004). Assembling the senses: Towards the design of cooperative

interfaces for visually impaired users.

Proceedings of Association for Computing Machin-

ery Conference on Computer Supported Cooperative Work

References

195

This page intentionally left blank

CHAPTER

Voice User Interfaces

Susan L. Hura

A voice user interface (VUI) is the script to a conversation between an automated

system and a user. This script contains all the utterances that the automated sys-

tem will speak to the user and the logic to decide which utterances to speak in

response to user input. Underlying the voice user interface is speech recognition

technology that has the ability to capture and decode the user’s spoken input to

allow the system to “understand” what the user has said. The majority of these

automated conversations between a user and a VUI take place over the phone,

as most speech technology is currently deployed in speech-enabled interactive

voice response interfaces (see Chapter 7). Increasingly, VUIs are being added

to the user experience of mobile and handheld devices, in-vehicle navigation

systems, and desktop computer applications.

Users come into automated conversations with a set of expectations about

how spoken conversation should work and the appropriate way to behave as a

cooperative speaker and listener. The overwhelming majority of users’ experience

comes from unscripted human-to-human speech, a feat that is far outside the cap-

abilities of today’s speech technology. This limitation is largely unknown to most

users, a great many of whom approach VUIs with the

Star Trek

model in mind: To

such users, speech-enabled systems are an always-listening intelligence awaiting

one’s every command. What Captain Picard requests is always understood, and

his spoken interactions with “Computer” appear effortless. Therefore, it is no sur-

prise that users’ expectations are somewhat out of sync with the way today’s VUIs

actually behave (Hura, 2005).

The expectation mismatch is particularly problematic because it leads users to

produce unpredictable speech input that is often handled poorly by speech sys-

tems. In this chapter, I offer my experiences as an evaluator and designer of VUIs.

The unifying theme for the chapter is the expectations mismatch and how to min-

imize it. I also provide an overview of speech technology from the perspective of a

VUI designer with the goal of achieving effortless spoken interaction with auto-

mated systems.

6.1

AUTOMATED CONVERSATION: HUMAN

VERSUS MACHINE

Commercial applications of speech recognition are largely bottom-up systems that

achieve understanding via statistically based matching of the user’s spoken input

with stored acoustic models of speech sounds, words, and pronunciations (see

Section 6.2 for more detail). Human beings, by contrast, rely heavily on top-down

knowledge about meaning and context when recognizing speech (Massaro,

2006). That is, when human beings are listening to spoken language, we are not

trying to match the incoming speech signal against any possible word in the lan-

guage. Instead, we automatically and unconsciously expect to hear words that

“fit” into the overall conversational context. Context includes microlevel informa-

tion such as which phoneme combinations are permissible, rules about sentence

structure that determine what sort of word is likely to come next, and nonlinguis-

tic world knowledge that rules out nonsensical interpretations of unclear words.

Because we know how the world works, we know that “Banjo” is a highly unlikely

response to the question “What do you want for lunch?” no matter how much the

acoustic signal points us toward that interpretation. Automated speech recogni-

tion systems have no knowledge of the world and must rely solely on the acoustic

signal and the bits of intelligence built in by VUI designers.

Even without the benefit of human-like top-down knowledge, many speech

recognition systems perform quite well, and through clever VUIs appear to be car-

rying on their half of the conversation admirably. When interactions with auto-

mated speech systems go well, users assume that their interlocutor (the system)

is conversing in the same manner they are—performing the tasks of conversation

in a way similar to how they do it and playing by the same rules of conver-

sation. Grice (1975) defined a set of maxims of conversation that describe the

unconscious rules by which we participate in conversations. According to the

maxims of conversation, we expect utterances to be of high quality (truthful and

honest), to offer an appropriate quantity of information, to be relevant to the con-

versation, and to be spoken in a clear, unambiguous manner. Automated speech

systems often do not follow these cooperative principles—accidentally or by

design—yet users tend to give the systems the benefit of the doubt and assume

that they are following the rules. This misapprehension on the part of users is

the fundamental challenge in designing effective VUIs.

Why do users expect so much of automated speech systems? In part it is

because VUIs sound very real. Most commercia l speech applications rely on a

6 Voice User Interfaces

198

set of recorded promp ts, spoken by a voice actor who has been careful ly

coached to produce each utterance as if i twerepartofanongoingconversation

(see additional information on prompts in Section 6.1.3). The bigger reason

behind users’ high expectations is the vast experience they have with spoken

language. Language is a hugely complex system governed by myriad rules, but

people are largely unaware of their own linguistic prowess. Human beings are

constantly immersed in spoken language. Exposure begins b efore birth and

there is evidence that human infants come into the world with the ability and

experience to begin immediately making sense of their linguistic environment.

Language is acquired automatically by human infants through simple exposure

and requires no adult intervention, other than simply conversing with the child.

Language is often characterized as an innate ability of human beings (Pinker,

1994; Reeves & Nass, 1996); thus, it is difficult for many people to understand

why simple conversation can be so daunting for automated systems. Talking

and listening are overlearned tasks for humans—we converse effortlessly,

moving our vocal musculature to produce speech sounds arranged into fluent

utterances without conscious effort. It’s

so easy

for us—why is it so hard for

computers?

Another reason that users expect a lot from speech systems is that they view

the ability to engage in conversation as evidence of the inherent intelligence of

the speech system. VUIs often seem smart, friendly, and even human—they

understand what you say and respond in ways that a person might in that circum-

stance. When users discuss their experiences with VUIs, they are often somewhat

disconcerted when they notice themselves attributing personal characteristics

(smart, friendly) to what they know to be a computer system. These embarrassed

users do not believe they are actually talking to a person, but they cannot help

talking about the VUI as “she” rather than “it.”

When users think this way, they are succumbing to attribution theory

(Heider, 1958). Attribution theory describes the phenomenon of viewing an indi-

vidual’s actions as evidence of his or her inherent characteristics. That is, we

see the actions of others as evidence of stable factors that are part of their person-

alities. Human beings have a strong propensity to overattribute individual beha-

viors to the inherent character of an individual. The pull of attribution theory is

so intense and so unconscious that we attribute personalities even to nonhumans

with whom we interact, such as computers and VUIs. Research by Nass and col-

leagues (Reeves & Nass, 1996; Nass & Brave, 2005) demonstrates that we tend to

interact with nonhuman actors according to the same social principals we apply

in human-to-human interactions. The fundamental attribution error is relevant

to VUIs in that users tend to see every action of a VUI as evidence of the person-

ality and abilities of the speech system. When a VUI offers relevant information

and gives appropriate responses to users, they tend to attribute this to the intelli-

gence and sensibility of the system. When a VUI says please and thank you, we

believe that the system is polite and friendly.

6.1 Automated Conversation: Human versus Machine

199

6.1.1 VUI Persona

Voice user interface designers often capitalize on users’ tendency to attribute

characteristics based on behavior in order to attach a particular

persona

to the

VUI. Here,

persona

is not used to describe user characteristics (Pruitt & Adlin,

2006) in the more familiar sense of the term in the general human factors com-

munity. Instead, in the world of speech technology,

persona

refers to the person-

ality of a speech interface inferred by users based on the behavior of the VUI.

Factors that contribute to the perception of persona are the style and tone of

prompts, the flow of dialog, and the responsiveness of the system. Some VUI

designers conceive of persona as an imaginary customer service representative

being emulated by the VUI. The concept of persona has been a continuing source

of controversy in the VUI community. VUI designers disagree vigorously on the

importance of persona in overall VUI design (Klie, 2007; Rolandi, 2007). Some

designers deny that persona exists at all, and others suggest that persona is the

most significant characteristic of an application.

The reality of persona almost certainly lies somewhere between these two

extremes. Attribution theory suggests that it may be impossible for users

not

infer a persona when participating in a spoken conversation, whether with a per-

son or with an automated system. Part of the human language faculty involves the

process of sizing up your interlocutor based on the way she speaks. This is not

something that humans can turn on and off; thus, it is impossible for there to be

“no persona” in a VUI. VUIs supposedly designed with no persona tend to sound

either schizophrenic (as if a variety of individuals are speaking rather than one

coherent voice), robotic (so devoid of normal human speech patterns that they

remind you of bad 1970s science fiction), or both. The relationship of persona to

VUI is akin to the relationship of color to the overall GUI design of a web page (Hura,

Polkosky, & Gilbert, 2006). Color is not the single defining quality of users’ percep-

tions of the web page, but their impressions are affected in significant ways by color.

There is also evidence that persona and the usability of a VUI are tightly cou-

pled. Users attribute persona through evidence they observe in the voice of the sys-

tem (e.g., pitch, pitch range, loudness, loudness range, articulatory precision), the

linguistic style of prompts (e.g., word choice, syntax, content, and structure of error

recovery prompts), and the overall organization of a VUI. Research by Polkosky

(2005) suggests that these factors predict an impression of dialog efficiency, inter-

active ease, and affective response in the user. There is strong similarity here to

the characteristics of general usability presented in the standard ISO 9241:11.

6.1.2 Jekyll and Hyde Personas

Because the tendency to attribute a personality to a VUI is so strong, one might

think that changing a user’s views on this personality would be difficult. Instead,

6 Voice User Interfaces

200

users’ perceptions of a VUI are notoriously fragile. One bad turn of the conversa-

tion and users can lose all confidence in the system as an intelligent actor in the

conversation. Speech recognition failures can cause this erosion of confidence,

but this is not always the case. Similarly, prompts that elicit unexpected responses

can, but do not always, cause user perceptions to change significantly. The exam-

ples that follow demonstrate these two classes.

The first case is when the user speaks an appropriate response clearly and

audibly, but the system does not recognize it. Users have a good sense of how

intelligible their utterances are, and when a speech system fails to recognize

well-formed responses users can become frustrated. The frustration can be exa-

cerbated by long and generic error prompts that do not allow for quick repair of

the conversation. As an example, consider the following:

YSTEM: Please select one of the following: account balances, transaction

history, transfers ...

SER: Account balances.

YSTEM: I’m sorry. I did not understand your response. Please choose

one of the foll owing: You can say: account balances, transaction

history ...

This error prompt belabors the recognition failure and draws the user’s attention

to it.

In fact, in human-to-human conversation this sort of misunderstanding is

common, and conversations are repaired and continued with only minimal inter-

ruption. The difference is that humans use brief conversational repair techniques

rather than taking a whole step back in the dialog. If a friend asks, “Where are

you going on vacation?” and fails to understand your response, the friend will

likely say, “Where?” as a follow-up rather than repeating the original question in

its entirety. Well-designed VUIs mimic conversational repair by using quick

reprompts that allow the conversation to get back on track quickly.

The second case is when a prompt elicits unexpected responses. The system

asks an open-ended question, and based on previous good experience with the

VUI, the user responds appropriately. The system, however, is incapable of deal-

ing with the response and goes into error prompting. This is essentially a case of

fooling the user into thinking the VUI is smarter than it really is (Rolandi, 2002).

The typical situation here is that a user has been successfully interacting with a

simple menu-based VUI for several turns of the conversation (the systems offers

a number of discrete choices, and the user responds by speaking one of these

choices). The user is then confronted by a more open-ended prompt, like the

one in the following example. Because the previous interaction has been so suc-

cessful, the user attributes intelligence and cooperation to the system, and cooper-

atively answers the question being asked.

6.1 Automated Conversation: Human versus Machine

201