Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Подождите немного. Документ загружается.
4.8.3 Changing Economics
Historical experience has led practitioners in computer-related technologies to
assume continuous improvements in performance and decreases in cost. There
have not yet been such trends for technologies supporting whole-body interfaces.
The prices of motion trackers, immersive displays, and motion platform output
devices have not dropped much in the last 10 years. Since 3D graphics hardware
was adopted by mass-market video games, it has become cheap, widely available,
and very high performance. As of this writing high-end tracker systems still have
five- to six-digit prices (in U.S. dollars), but Nintendo’s newest console, Wii, is a
mass-market success.
The Wii game console includes handheld controllers that sense hand acceler-
ation and, when pointed at the TV screen, also sense hand orientation. In the
game Wii-Sports Tennis, the sensed acceleration is used to control the motion of
the racquet (Figure 4.21). The orientation feature enables users to point at objects
FIGURE
4.21
Nintendo Wii game system.
Wii includes rudimentary motion tracking in the handheld controllers. A player
swings a virtual tennis racket by swinging her hand and arm. (Courtesy of the
Department of Computer Science, UNC at Chapel Hill.)
4 Locomotion Interfaces
142
displayed on the TV. The Wii-Fit game includes a platform that reports the weight
and center-of-balance position of the user who is standing on it.
While we have not yet seen any whole-body locomotion interfaces in mass-
market video games, we suspect they will soon appear. We ourselves are currently
developing interfaces that use Wii controllers. We are hopeful that Wii will be the
breakthrough product that will, in a reasonably short time, result in widely avail-
able, cheap, and high-performance whole-body trackers for all applications and in
particular for virtual locomotion.
REFERENCES
Arthur, K. (2000). Effects of Field of View on Performance with Head-Mounted Displays.
CS Technical Report TR00-019. Chapel Hill: Department of Computer Science, University
of North Carolina.
Berthoz, A. (2000).
The Brain’s Sense of Movement
. Weiss, G., trans. Cambridge, MA:
Harvard University Press.
Bouguila, L., Evequoz, F., Courant, M., & Hirsbrunner, B. (2004). Walking-pad: A step-in-
place locomotion interface for virtual environments.
Proceedings of Sixth International
Conference on Multimodal Interfaces
. State College, PA, ACM, 77–81.
Bowman, D. A., Koller, D., & Hodges, L. F. (1997). Travel in immersive virtual environ-
ments: An evaluation of viewpoint motion control techniques.
Proceedings of IEEE Vir-
tual Reality Annual International Symposium
, 45–52.
Brooks, F. P. (1988). Grasping reality through illusion—interacti ve graphics serving science.
Proceedings of SIGCHI Conference on Human Factors in Computing Systems
, Washing-
ton, DC, ACM, 1-11.
Cobb, S. V. G., & Nichols, S. C. (1998). Static posture tests for the assessment of postural
instability after virtual environment use.
Brain Research Bulletin
47(5):459–64.
Darken, R. P., Cockayne, W. R., & Carmein, D. (1997). The omni-directional treadmill: A
locomotion device for virtual worlds.
Proceedings of Association for Computing Machin-
ery User Interface Software and Technology
. Banff, Canada, ACM, 213–21.
Dichgans, J., & Brandt, T. (1977). Visual–vestibular interaction: Effects on self-motion per-
ception and postural control.
Perception
8:755–804.
Draper, M. (1998). The Adaptive Effects of Virtual Interfaces: Vestibulo-Ocular Reflex and
Simulator Sickness. PhD diss., University of Washington, Seattle.
Duh, H. B.-L., Parker, D. E., Phillips, J., & Furness, T. A. (2004). “Conflicting” motion cues at
the frequency of crossover between the visual and vestibular self-motion systems evoke
simulator sickness.
Human Factors
46:142–53.
Fleming, S. A., Krum, D. M., Hodges, L. F., & Ribarsky, W. (2002). Simulator sickness
and presence in a high field-of-view virtual environment [abstract].
Proceedings of
Computer–Human Interaction, Human Factors in Computing Systems
.Minneapolis,
ACM, 784–85.
Foxlin, E. (2002). Motion tracking requirements and technologies. In Stanney, K. M., ed.
Handbook of Virtual Environments
. Mahwah, NJ: Erlbaum, 163–210.
Gregory, R. L. (1970).
The Intelligent Eye
. New York: McGraw-Hill.
References
143
Harris, L., Jenkin, M., & Zikovitz, D. C. (1999). Vestibular cues and virtual environments:
Choosing the magnitude of the vestibular cue.
Proceedings of IEEE Virtual Reality Confer-
ence
, Houston, 229–36.
Hollands, M. A., Patla, A. E., & Vickers, J. N. (2002). “Look where you’re going!”: Gaze beha-
viour associated with maintaining and changing the direction of locomotion.
Experimen-
tal Brain Research
143(2):221–30.
Hollerbach, J. M. (2002). Locomotion interfaces. In Stanney, K. M., ed.,
Handbook of Virtual
Environments
. Mahwah, NJ: Erlbaum, 239–54.
Hollerbach, J. M., Christensen, R. R., Xu, Y., & Jacobsen, S. C. (2000). Design specifications
for the second generation Sarcos Treadport locomotion interface.
Proceedings of ASME
Dynamic Systems and Control Division, Haptics Symposium
, Orlando, 1293–98.
Howard, I. P. (1986). The vestibular system. In Boff, K. R., Kaufman, L., & Thomas, J. P.,
eds.,
Handbook of Perception and Human Performance
, 1:11–26. New York: Wiley-
Interscience.
Inman, V. T., Ralston, H. J., & Frank, T. (1981).
Human Walking
. Baltimore/London:
Williams and Wilkins.
Insko, B. (2001). Passive Haptics Significantly Enhances Virtual Environments. CS Technical
Report TR01-017. Chapel Hill: University of North Carolina, Department of Computer
Science.
Iwata, H. (1999). Walking about virtual environments on an infinite floor.
Proceedings of
IEEE Virtual Reality Conference
, Houston, 286–95.
Iwata, H., & Yoshida, Y. (1999). Path reproduction tests using a Torus treadmill.
Presence:
Teleoperators and Virtual Environments
8(6):587–97.
Kennedy, R. S., Drexler, J. M., Compton, D. E., Stanney, K. M., Lanham, D. S., & Harm,
D. L. (2003). Configural scoring of simulator sickness, cybersickness and space adaptation
syndrome: Similarities and differences. In Hettinger, L. J., & Haas, M. W., eds.,
Virtual
and Adaptive Environments: Applications, Implications, and Human Performance
.
Mahwah, NJ: Erlbaum, 247–78.
Kennedy, R. S., & Lilienthal, M. G. (1995). Implications of balance disturbances following
exposure to virtual reality systems.
Proceedings of IEEE Virtual Reality Annual Interna-
tional Symposium
, Research Triangle Park, NC, 35–39.
Kolasinski, E. (1995). Simulator Sickness in Virtual Environments. Technical Report
1027. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.
Lackner, J. R. (1977). Induction of illusory self-rotation and nystagmus by a rotating
sound-field.
Aviation, Space, and Environmental Medicine
48(2):129–31.
LaViola Jr., J. J. (2000). A discussion of cybersickness in virtual envronments.
SIGCHI
Bulletin
32(1):47–55.
LaViola Jr., J. J., Feliz, D. A., Keefe, D. F., & Zeleznik, R. C. (2001). Hands-free multi-scale
navigation in virtual environments.
Proceedings of Association for Computing Machinery
Symposium on Interactive 3D Graphics
, Research Triangle Park, NC, 9–15.
Liu, G., Zhang, J., Wang, W., & McMillan, L. (2005). A system for analyzing and indexing
human-motion databases.
Proceedings of Association for Computing Machinery SIGMOD
International Conference on Management of Data
, Baltimore, 924–26.
Martini, F. (1999).
Fundamentals of Anatomy and Physiology
, 4th ed. Upper Saddle River,
NJ: Prentice Hall.
4 Locomotion Interfaces
144
Meehan, M., Razzaque, S., Insko, B., Whitton, M. C., & Brooks, F. P. J. (2005). Review of four
studies on the use of physiological reaction as a measure of presence in a stressful virtual
environment.
Applied Psychophysiology and Biofeedback
30(3):239–58.
Meehan, M., Razzaque, S., Whitton, M., & Brooks, F. (2003). Effects of latency on presence
in stressful virtual environments.
Proceedings of IEEE Virtual Reality
, Los Angeles,
141–48.
Mine
´
, M. R. (1997). Exploiting Proprioception in Virtual-Environment Interaction. CS Tech-
nical Report TR97-014. Chapel Hill: University of North Carolina, Department of Com-
puter Science.
Mohler, B., Thompson, W. B., Creem-Regehr, S., Pick, H. L., Warren, W., Rieser, J. J., &
Willemsen, P. (2004). Virtual environments I: Visual motion influences locomotion in a
treadmill virtual environment.
Proceedings of First Symposium on Applied Perception
in Graphics and Visualization
. Los Angeles, 19–22.
Nitzsche, N., Hanebeck, U. D., & Schmidt, G. (2004). Motion compression for telepresent
walking in large target environments.
Presence: Teleoperators and Virtual Environments
13(1):44–60.
Pang, T. K., Lim, K. Y., & Quek, S. M. (2000). Design and development of a stereoscop ic
display rig for a comparative assessment of r emote freight handling per formance.
Proceedings of Joint Conference o f APCHI and AS EAN Ergono mics
, Singap ore, 62– 67.
Peterson, B., Wells, M., Furness, T., & Hunt, E. (1998). The effects of the interface on navi-
gation in virtual environments.
Proceedings of Human Factors and Ergonomics Society
Annual Meeting
, 1496–505.
Razzaque, S. (2005). Redirected Walking. CS Technical Report TR05-018. Chapel Hill: Uni-
versity of North Carolina, Department of Computer Science.
Razzaque, S., Kohn, Z., Whitton, M. (2001). Redirected walking.
Proceedings of Euro-
graphics Workshop
, 289–94.
Riecke, B. E., van Veen, H. A. H. C., & Bulthoff, H. H. (2002). Visual homing is possible
without landmarks: A path integration study in virtual reality.
Presence: Teleoperators
and Virtual Environments
11(5):443–73.
Robinett, W., & Holloway, R. (1992). Implementation of flying, scaling and grabbing in vir-
tual environments.
Proceedings of Association for Computing Machinery Symposium on
Interactive 3D Graphics
, Cambridge, MA, 189–92.
Rolfe, J. M., & Staples, K. J. (Eds.) (1988).
Flight Simulation.
Cambridge: Cambridge Univer-
sity Press.
Schell, J., & Shochet, J. (2001). Designing interactive theme park rides.
IEEE Computer Gra-
phics and Applications
21(4):11–13.
Slater, M., & Steed, A. (2000). A virtual presence counter.
Presence: Teleoperators and Vir-
tual Environments
9(5):413–34.
Slater, M., & Usoh, M. (1993). Presence in immersive virtual environments.
Proceedings of
IEEE Virtual Reality Annual International Symposium
, 90–96.
Slater, M., & Usoh, M. (1994). Body-centered interaction in immersive virtual environ-
ments. In Mageneat-Thalman, N., & Thalmann, D., eds.,
Artifcial Life and Virtual Reality
.
New York: John Wiley and Sons, 125–48.
Slater, M., Usoh, M., & Steed, A. (1995). Taking steps: The influence of a walking technique
on presence in virtual reality.
ACM Transactions on Computer–Human Interaction
2(3):201–19.
References
145
Stoakley, R., Conway, M. J., & Pausch, R. (1995). Virtual reality on a WIM: Interactive
worlds in miniature.
Proceedings of Human Factors and Computing Systems
, 265–72.
Stoffregen, T. A., Hettinger, L. J., Haas, M. W., & Smart, L. J. (2000). Postural instability and
motion sickness in a fixed-base flight simulator.
Human Factors
42:458–69.
Templeman, J. N., Denbrook, P. S., & Sibert, L. E. (1999). Virtual locomotion: Walking in
place through virtual environments.
Presence: Teleoperators and Virtual Environments
8(6):598–617.
Templeman, J. N., Sibert, L., Page, R. C., & Denbrook, P. S. (2007). Pointman—A device-
based control for realistic tactical movement.
Proceedings of IEEE Symposium on 3D
User Interfaces
, Charlotte, NC.
Usoh, M., Arthur, K., Whitton, M. C., Bastos, R., Steed, A., Slater, M., & Brooks, F. P. (1999).
Walking > walking-in-place > flying in virtual environments.
Proceedings of SIGGRAPH
.
Los Angeles, 359–64.
Warren Jr., W. H., Kay, B. A., Zosh, W. D., Duchon, A. P., & Sahuc, S. (2001). Optic flow is
used to control human walking.
Nature Neuroscience
4(2):213–16.
Yan, L., Allison, R. S., & Rushton, S. K. (2004). New simple virtual walking method—walking
on the spot. Paper presented at Ninth Annual Immersive Projection Technology (IPT)
Symposium.
Young, L. R., Adelstein, B. D., & Ellis, S. R. (2006). Demand characteristics of questionnaire
used to assess motion sickness in a virtual environment.
Proceedings of IEEE Virtual
Reality
, Alexandria, VA, 97–102.
4 Locomotion Interfaces
146
5
CHAPTER
Auditory Interfaces
S. Camille Peres, Virginia Best, Derek Brock, Barbara
Shinn-Cunningham, Christopher Frauenberger,
Thomas Hermann, John G. Neuhoff, Louise
Valgerður Nickerson, Tony Stockman
Auditory interfaces are bidirectional, communicative connections between two
systems—typically a human user and a technical product. The side toward the
machine involves machine listening, speech recognition, and dialog systems.
The side toward the human uses auditory displays. These can use speech or pri-
marily nonspeech audio to convey information. This chapter will focus on the
nonspeech audio used to display information, although in the last section of
the chapter, some intriguing previews into possible nonspeech audio-receptive
interfaces will be presented.
Auditory displays are not new and have been used as alarms, for communica-
tion, and as feedback tools for many decades. Indeed, in the mid-1800s an auditory
display using Morse code and the telegraph ushered in the field of telecommu-
nications. As technology has improved, it has become easier to create auditory dis-
plays. Thus, the use of this technology to present information to users has become
commonplace, with applications ranging from computers to crosswalk signals
(Brewster, 1994; Massof, 2003). This same improvement in technology has coinci-
dentally increased the
need
for auditory displays.
Some of the needs that can be met through auditory displays include (1) pre-
senting information to visually impaired people, (2) providing an additional infor-
mation channel for people whose eyes are busy attending to a different task,
(3) alerting people to error or emergency states of a system, and (4) providing
information via devices with small screens such as PDAs or cell phones that have
a limited ability to display visual information. Furthermore, the auditory system
is well suited to detect and interpret multiple sources and types of information
as will be described in the chapter.
Audio displays (or auditory interfaces) as they are experienced now, at the
beginning of the 21st century, are more sophisticated and diverse than the bells
and clicks of the past. As mentioned previously, this is primarily due to the
increasing availability of powerful technical tools for creating these displays. How-
ever, these tools have only recently become accessible to most engineers and
designers, and thus the field of auditory displays is somewhat in its adolescence
and for many is considered relatively new. Nevertheless, there is a substantial
and growing body of work regarding all aspects of the auditory interface. Because
this field is young and currently experiencing exponential growth, the lexicon and
taxonomy for the various types of auditory displays is still in development. A dis-
cussion of the debates and nuances regarding the development of this taxonomy is
not appropriate for this chapter, but the interested reader will find details on this
topic in Kramer’s book on auditory displays (1994), distributed by the Interna-
tional Community on Auditory Displays (ICAD) (
www.icad.org
), as well as other
sonification sites (e.g.,
http://sonification.de
).
The terms used in this chapter are outlined and defined in the following sec-
tions. We have organized the types of auditory displays primarily by the method
or technique used to create the display. Additionally, all of the sound examples
are available at
www.beyondthegui.com
.
Sonification of Complex Data
Sonification is the use of nonspeech sound to render data, either to enhance the
visual display or as a purely audio display. Sonification is routinely used in hospi-
tals to keep track of physiological variables such as those measured by electrocar-
diogram (ECG) machines. Audio output can draw the attention of medical staff to
significant changes in patients while they are otherwise occupied. Other sonifica-
tions include the rhythms in electroencephalogram (EEG) signals that can assist
with the prediction and avoidance of seizures (Baier, Hermann, Sahle, & Ritter,
2006) and sonification of the execution of computer programs (Berman & Gallagher,
2006). There are various types of sonifi cation techniques, and these will be
elaborated throughout the chapter and particularly in Section 5.2.4.
Considerable research has been done regarding questions that arise in design-
ing effective mappings for sonification of data. Among the key issues are
voicing
,
property
,
polarity
, and
scaling/context
.
Voicing
deals with the mapping of data
to the sound domain; that is, given a set of instruments, which one should be
associated with which variable?
Property
deals with how changes in a variable should be represented, such as
should changes in a data variable be mapped to pitch, amplitude, or tempo?
Polar-
ity
deals with the way a property should be changed (Walker, 2002), that is, should
a change in a variable cause the associated sound property to rise or fall? Suppose
weight is mapped to tempo: Should an increase in weight lead to an increase in
tempo, or a decrease given that increasing weight generally would be associated
with a slower response?
5 Auditory Interfaces
148
Scaling
deals with how quickly a sound property should change. For instance,
how can we convey an understanding of the absolute values being displayed,
which requires establishing maximum and minimum values displayed, where
the starting value is in the scale, and when the data cross zero? How much of
a change in an original data variable is indicated by a given change in the
corresponding auditory display parameter?
Audification
A very basic type of auditory display, called
audification
, is simply presenting
raw data using sound. This will be described in more detail in Section 5.2.4, but
essentially everything from very-low-frequency seismic data (Hayward, 1994) to
very-high-frequency radio telescope data (Terenzi, 1988) can be transformed into
perceptible sound. Listeners can often derive meaningful information from the
audification.
Symbolic/Semantic Representations of Infor mation
Similar to other types of interfaces, auditory interfaces or displays are often
needed for a task other than data analysis or perceptualization. For instance, GUIs
use icons to represent different software programs or functions within a program.
This type of visual display is more semantic in nature and does not require
analysis on the part of the user.
The auditory equivalents of visual icons are auditory displays known as
audi-
tory icons
and
earcons
. These displays are very useful in translating symbolic
visual artifacts into auditory artifacts and will be discussed several times through
the chapter. An example of an auditory icon is the paper-“crinkling” noise that is
displayed when a user empties the Trash folder. This technique employs sounds
that have a direct and thus intuitive connection between the auditory icon and
the function or item.
Earcons are a technique of representing functions or items with more abstract
and symbolic sounds. For example, just as Morse code sound patterns have an
arbitrary connection to the meaning of the letter they represent, the meaning of
an earcon must be learned. These types of displays can be particularly appropriate
for programs that have a hierarchical structure as they allow for the communica-
tion of the structure in addition to the representation of the functions.
A new symbolic/semantic technique for auditory displays that shows some
promise is the use of
spearcons
. These are nonspeech cues used in the way that
icons or earcons would be. They are created by speeding up a spoken phrase until
it is not recognized as speech (Walker, Nance, & Lindsay, 2006). This representa-
tion of the spoken phrase, such as a person’s name in a phone list, can be slowed
down to a recognizable level to facilitate learning the association between the
spearcon and the name. Once this association is made, the spearcon can be played
using the shortest duration to reduce the amount of time necessary to present the
auditory display.
5 Auditory Interfaces
149
It is very important to understand, and should be clear after reading the entire
chapter, that these techniques are not normally and sometimes not even ideally
used exclusively. Furthermore, they are not necessarily independent of each
other. For instance, in daily weather sonifications (Hermann, Drees, & Ritter,
2005), most data variables were displayed via parameterized auditory icons (e.g.,
water sounds gave an iconic link to rain, and the duration of the sound allowed
the user to judge the amount of rain per unit of time). When designing and testing
auditory displays, designers can consider three different dimensions or axes:
the interpretation level, from analogic to symbolic; interactivity, from noninterac-
tive to tightly closed interaction; and “hybridness,” from isolated techniques to
complex mixtures of different techniques. The ability to use these three dimen-
sions gives the designer a wide and intriguing range of tools to better meet users’
needs.
5.1
NATURE OF THE INTERFACE
It is important for those who will design and test auditory displays to understand
some of the basic perceptual properties of the human auditory system. These
properties dictate how humans experience and interpret sounds, and this section
describes some of the more fundamental elements of the auditory perception
process.
5.1.1 Basic Properties of Sound
Sound arises from variations in air pressure caused by the motion or vibration of
an object. Sounds are often described as pressure variations as a function of time
and are plotted as a waveform. Figure 5.1 shows the waveform for a pure sinusoid,
which is periodic (repeats the same pattern over and over in time) and is charac-
terized by its frequency (number of repetitions per second), amplitude (size of the
pressure variation around the mean), and phase (how the waveform is aligned in
time relative to a reference point). All complex sounds can be described as the
sum of a specific set of sinusoids with different frequencies, amplitudes, and
phases.
Sinusoids are often a natural way to represent the sounds we hear because the
mechanical properties of the cochlea break down input sounds into the compo-
nents at different frequencies of vibration. Any incoming sound is decomposed
into its component frequencies, represented as activity at specific points along
the cochlea. Many natural sounds are periodic (such as speech or music) and con-
tain energy at a number of discrete frequencies that are multiples of a common
(fundamental) frequency. Others are nonperiodic (such as clicks or white noise)
and contain energy that is more evenly distributed across frequency.
5 Auditory Interfaces
150
5.1.2 Human Sensitivity to Auditory Dimensions
The basic properties of sound give rise to a number of perceptual dimensions that
form the basis of auditory displays. The effectiveness of an auditory display
depends critically on how sensitive listeners are to changes along the dimensions
used to represent the relevant information. The following sections present a brief
examination of human sensitivity to the dimensions of frequency and pitch, loud-
ness, timbre, and spatial location. (For an extensive discussion, see Moore, 2003.)
Frequency and Pitch
As mentioned before, sound frequency is a fundamental organizing feature of
the auditory system and a natural dimension for carrying information in auditory
displays. Effective use of the frequency domain must take into account both
the range of frequencies to which human ears respond and how well listeners
can distinguish neighboring frequencies within this range.
Human listeners are able to detect sounds with frequencies between about
16 Hz and 20 kHz, with sensitivity falling off at the edges of this range. Fre-
quency resolution (i.e., the ability to distinguish different frequencies) within
the audible range is determined by the peripheral auditory system, which acts
like a series of overlapping filters. Each filter is responsive to the sound energy
Period T = 1/frequency
1
0
0T
Time
2T
Amplitude
−1
FIGURE
5.1
The waveform for a pure sinusoid.
The elements of a waveform are frequency (number of repetitions per second),
amplitude (size of the pressure variation around the mean), and phase (how the
waveform is aligned in time relative to a reference point).
5.1 Nature of the Interface
151