Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Подождите немного. Документ загружается.
3.6.2 Technical Tests
Once the interface has been implemented, it is important to conduct a series of
technical performance tests. An interface that responds in a delayed fashion or
is imprecise will not be successful. Interfaces that assume slow or fairly static
movements are less sensitive to this than those in which movement is allowed
to be fluent and natural.
Precision of Position, Orientation, and Timing
The important precision parameters that should be tested follow:
Position:
Measure any discrepancy between where the system thinks the gesture
is performed and where it really is in real-world units.
Orientation:
When applicable, measure any discrepancy between the direction
that a gesture is pointing and the real direction.
Timing:
Measure the response delay in time units. Note start, stroke, and end
times, where stroke is the climax of the gesture. For example, the stroke of rais-
ing a hand in the air is when the hand reaches its highest position.
Evaluate whether the results are critical to the application and whether it is safe to
have the user compensate by using a given feedback. This feedback could be
sound to mark the moment that the application triggers an action, or a visual
marker that shows where the application thinks the fingers are.
Accuracy of Classifications
This parameter is a measure of gesture detection robustness. Accuracy is the per-
centage of gestures detected by the computer based on a large series of gesture
classifications.
3.7
CASE STUDY
A case study for a gesture interface can be found at
www.beyondthegui.com
.
3.8
SUMMARY
A gesture interface can be a good way to enhance the immersive feeling in a
virtual-reality application and a good means of making the user’s experience feel
more natural. This chapter provided background information on various work
and types of gesture interfaces. It also outlined fundamental theory and issues
to be addressed when considering and designing such an interface.
Gestures are particularly useful when relating to the space around the user as
well as for conveying commands. In particular, we stressed that the gesture
3 Gesture Interfaces
102
vocabulary must be concise, intuitive, and ergonomic to perform, because it can be
tiring to use such an interface for a prolonged time.
A procedure for achieving this was proposed and demonstrated, and the
resulting gesture vocabulary was tested on various user groups and compared to
a purely technical-based gesture vocabulary. The success of the gesture vocabu-
lary depended on the user groups, and the technical gestures were not as easy
to use as the intuitive set.
3.9
FUTURE TRENDS
The future will likely bring many interesting breakthroughs in this area. Raw com-
puter vision and other detection systems will evolve into robust classifiers with
less obstructive designs. This means that there will be no devices attached to
the user, so that the user can feel completely immersed in the virtual environ-
ment that blends perfectly with the real world.
Classical computer systems see input modes and output modes as interchangeable
input modules to the system. This defines a clear boundary between the user and the
system. We should stop thinking of an interface as just ... an interface. We do not say
that we interface with a car. We say that we drive to some location, that is, the task of
driving to the location is in focus. If we can bring the application and the user closer to
each other in an invisible interface, the user focuses on the objective, not the computer
or interface, which is just a natural part of the environment.
We predict that hands-free gesture interfaces will be as smooth as Jefferson’s
(Han, 2005) multitouch interface, and will be used to create intelligent rooms with
ubiquitous computing and advanced immersive virtual- and mixed-reality envir-
onments, which will be aware of people in the real world. These environments
will be aware not only of people’s presence but of their emotions, actions, and
intentions as well. Computers will be able to adapt to users to such a degree that
they become a “natural” part of life—almost like a living organism.
REFERENCES
Agliati, A., Vescovo, A., & Anolli, L. (2006). A new methodological approach to nonverbal
behavior analysis in cultural perspective.
Behavior Research Methods
38(3):364–71.
Aish, F., Broll, W., Stoerring, M., Fatah, A., & Mottram, C. (2004). ARTHUR: An augmented
reality collaborative design system.
Proceedings of First European Conference on Visual
Media Production
, London.
Aviles-Arriaga, H. H., Sucar, L. E., & Mendoza, C. E. (2006). Visual recognition of similar
gestures.
Proceedings of Eighteenth International Conference on Pattern Recognition
1:1100–103.
Beringer, N. (2002). Evoking gestures in SmartKom—Design of the graphical user interface.
Lecture Notes in Artificial Intelligence
2298, 228–40.
3.9 Future Trends
103
Betke, M., Gips, J., Fleming, P. (2002). The camera mouse: visual tracking of body features
to provide computer access for people with severe disabilities.
IEEE Transactions on
Neural Systems and Rehabilitation Engineering
10(1):1–20.
Bevan, N., & Curson, I. (1997). Methods for measuring usability.
Proceedings of Sixth Inter-
national Federation of Information Processing Conference on Human–Computer Interac-
tion
, Sydney, Australia.
Bowyer, K. W., Chang, K., & Flynn, P. 2006. A survey of approaches and challenges in 3D and
multi-modal 3D+2D face recognition.
Computer Vision and Image Understanding
101:1–15.
Caldwell, D. G., Tsagarakis, N., & Giesler, C. (1999). An integration tactile/shear feedback
array for simulation of finger mechanoreceptor.
Proceedings of 1999 IEEE International
Conference on Robotics and Automation
, Detroit.
Carbini, S., Delphin-Poulat, L., Perron, L., & Viallet, J. E. (2006). From a Wizard of Oz experi-
ment to a real-time speech and gesture multimodal interface.
Signal Processing
86:3559–77.
Cassell, J. (1998). A framework for gesture generation and interpretation. In Cipolla, R.,
& Pentland, A., eds.,
Computer Vision in Human–Machine Interaction
. New York:
Cambridge University Press, 191–215.
De Silva, P. R., Marasinghe, C. A., Madurapperuma, A. P., Lambacher, S. G., & Osano, M. (2006).
Modeling cognitive structure of emotion for developing a pedagogical agent based interactive
game.
IEEE International Conference on Systems, Man and Cybenetics
1(8–11):339–44.
Dzida, W., & Freitag, R. (1998). Making use of scenarios for validating analysis and design.
IEEE Transactions on Software Engineering
24:1182–96.
Eaton, C. (1997).
Electronic Textbook on Hand Surgery
. Available at
http://www.eatonhand.
com/
.
Federico, M. (1999).
Usability Evaluation of a Spoken Data-Entry Interface
. Povo, Trento,
Italy: ITC-Irst Centro per la Ricera Scientifica e Technologica.
Fihl, P., Holte, M. B., Moeslund, T. B, & Reng, L. (2006). Action recognition using motion
primitives and probabilistic edit distance.
Proceedings of Fourth International Conference
on Articulated Motion and Deformable Objects
. Mallorca, Spain.
Fingerworks. (2002). iGesture Pad. Available at
http://www.fingerworks.com/igesture.html
.
Freeman, W. T., & Weissman, C. D. (1995). Television control by hand gestures.
Proceed-
ings of IEEE International Workshop on Automatic Face and Gesture Recognition
.
Goto, S. (2006). The case study of an application of the system, “BodySuit” and “Robotic-
Music”: Its introduction and aesthetics.
Proceedings of Conference on New Interfaces
for Musical Expression,
Paris, 292–95.
Grant, C. (2002).
Ten Things You Should Know about Hand and Wrist Pain
. Ann Arbor, MI:
F-One Ergonomics.
Hall, J. (2006). Nonverbal behavior, status and gender: How do we understand their
relations?
Psychology of Women Quarterly
30:384–91.
Han, J. Y. (2005). Low-cost multi-touch sensing through frustrated total internal reflection.
Proceedings of Eighteenth Annual Association for Computing Machinery Symposium on
User Interface Software and Technology
, Seattle, 115–18.
Hedge, A., Muss, T. M., & Barrero, M. (1999).
Comparative Study of Two Computer Mouse
Designs
. Ithaca, NY: Cornell Human Factors Laboratory.
Hummels, C., & Stapers, P. J. (1998). Meaningful gestures for human computer interaction:
Beyond hand postures.
Proceedings of Third International Conference on Automatic Face
& Gesture Recognition
, Nara, Japan, 591–96.
3 Gesture Interfaces
104
Jaimes, A., & Sebe, N. (2005). Multimodal human computer interaction: A survey.
Proceed-
ings of IEEE International Workshop on Human–Computer Interaction in Conjunction
with International Conference on Computer Vision
, Beijing.
Keates, S., & Robinson, P. (1998). The use of gestures in multimodal input.
Proceedings of Asso-
ciation for Computing Machinery Special Interest Group on Accessible Computing
, 35–42.
Keir, P. J., Bach, J. M., & Rempel, D. M (1998). Effects of finger posture on carpal
tunnel pressure during wrist motion. San Francisco: Division of Occupational Medicine,
University of California.
Klein, J., Moon, Y., & Picard, R. W. (2002). This computer responds to user frustration:
Theory, design, and results.
Interacting with Computers
14(2):119–40.
Langdon, P., Keates, S., Clarkson, P. J., & Robinson, P. (2000). Using haptic feedback to
enhance computer interaction for motion-impaired users.
Proceedings of Third Interna-
tional Conference on Disability, Virtual Reality and Associated Technologies
, Alghero, Italy.
Lee, J., & Kunjii, T. (1995). Model-based analysis of hand posture.
IEEE Computer Graphics
and Applications
15:77–86.
Lin, J., Wu, Y., & Huang, T. S. (2001). Modeling the natural hand motion constraints.
Proceedings of Fifth Annual ARL Federated Laboratory Symposium
, 105–10.
Long, A. C., Landay, J. A., Rowe, L. A., & Michiels, J. (2000). Visual similarity of pen
gestures.
CHI Letters: Human Factors in Computing Systems (SIGCHI)
2(1):360–67.
Loomis, J. M., Klatzky, R. L., Philbeck, J. W., & Golledge, R. G. (1998). Assessing auditory
distance perception using perceptually directed action.
Perception & Psychophysics
60:966–80.
Min, B. W., Yoon, H. S., Soh, J., Ohashi, T., & Ejima, T. (1999). Gesture-based editing system
for graphic primitives and alphanumeric characters.
Engineering Applications of Artificial
Intelligence
12(4):429–41.
Moeslund, T. B., & Granum, E. (2003). Sequential Monte Carlo tracking of body parameters
in a sub-space.
IEEE International Workshop on Analysis and Modeling of Faces and
Gestures
, 84–91.
Moeslund, T. B., Sto
¨
rring, M., Broll, W., Aish, F., Liu, Y., & E. Granum. (2004). The
ARTHUR system: An augmented round table.
Proceedings of Eighth World Multi-
Conference on Systemics, Cybernetics and Informatics
, Orlando.
Moeslund, T. B., Sto
¨
rring, M., & Granum, E. (2004). Pointing and command gestures for
augmented reality.
Proceedings of FGnet Workshop on Visual Observation of Deictic
Gestures, Pointing, in Association with Seventeenth International Conference on Pattern
Recognition
, Cambridge.
Moustakas, K., Strintzis, M. G., Tzovaras, D., Carbini, S., Bernier, O., Viallet, J. E., Raidt , S.,
Mancas, M., Dimiccoli, M., Yagdi, E., Balci, S., & Leon, E. I. (2006). Masterpiece: Physical
interaction and 3D content–based search in VR applications.
IEEE Multimedia
13(6):
92–100.
Mu
¨
ller-Tomfelde, C., & Steiner, S. (2001). Audio-enhanced collaboration at an interactive
electronic whiteboard.
Proceedings of International Conference on Auditory Display
,
Espoo, Finland.
Nielsen, J. (1992). The usability engineering life cycle.
Computer
25:12–22.
Paggio, P., & Music, B. (2000). Linguistic interaction in staging—A language engineering
view. In Qvortrup, L., ed.,
Virtual Interaction: Interaction in/with Virtual Inhabited 3D
Worlds
. New York: Springer-Verlag, 235–49.
References
105
Pantic, M., & Patras, I. (2006). Dynamics of facial expression: Recognition of facial actions
and their temporal segments from face profile image sequences.
IEEE Transactions on
Systems, Man and Cybernetics, Part B
36(2):433–49.
Pantic, M., & Rothkrantz, L. J. M. (2003). Toward an affect-sensitive multimodal human–
computer interaction.
Proceedings of IEEE
91(9):1370–90.
Rasmussen, N. T., Sto
¨
rring, M., Moeslund, T. B., & Granum, E. (2006). Real-time tracking
for virtual environments using SCAAT Kalman filtering and unsynchronized cameras.
Proceedings of International Conference on Computer Vision Theory and Applications
,
Setu
´
bal, Portugal.
Riviera, J-B., & Guitton, P. (2004). Hand posture recognition in large-display VR environ-
ments. In
Gesture-Based Communication in Human-Computer Interaction
. Berlin:
Springer, 259–68.
Sage, K., Howell, A. J., & Buxton, H. (2003). Developing context sensitive HMM gesture rec-
ognition.
Proceedings of Fifth International Workshop on Gesture and Sign Language-
Based Human–Computer Interaction
, Genova, Italy.
Shaw, G., & Hedge, A. (2002). The effect of keyboard and mouse placement on shoulder
muscle activity and wrist posture.
Ergo Info
, available at
http://www.humanscale.com/
ergo_info/articles.cfm?type=studies
.
Steininger, S., Lindemann, B., & Paetzold, T. (2002). Labeling of gestures in SmartKom—
The coding system.
Proceedings of Gesture and Sign Language in Human–Computer
Interaction, International Gesture Workshop
, London, 215–27.
Sto
¨
rring, M., Andersen, H. J., & Granum, E. (2004). A multispectral approach to robust
human skin detection.
Proceedings of Second European Conference on Color in Graphics,
Images and Vision
.
Sto
¨
rring, M., Granum, E., & Moeslund, T. B. (2001). A natural interface to a virtual
environment through computer vision-estimated pointing gestures.
Proceedings of
Workshop on Gesture and Sign Language-Based HCI
, London.
Streitz, N. A., et al. (2001).
Roomware: Towards the Next Generation of Human–Computer
Interaction Based on an Integrated Design of Real and Virtual Worlds
. German National
Research Center for Information Technology, Integrated Publication and Information
Systems Institute.
Tsagarakis, N. G., Gray, J. O., Caldwell, D. G., Zannoni, C., Petrone, M., Testi, D.,
& Viceconti, M. (2006). A haptic-enabled multimodal interface for the planning of hip
arthroplasty.
IEEE Multimedia
13(6):40–48.
Tse, E., Shen, C., Greenberg, S., & Forlines, C. (2006). Enabling interaction with single user
applications through speech and gestures on a multi-user tabletop.
Proceedings of Work-
ing Conference on Advanced Visual Interfaces Table of Contents
, Venezia, Italy, 336–43.
Vroomen, J., & de Gelder, B. (2004). Temporal ventriloquism: Sound modulates the flash-
lag effect.
Journal of Experimental Psychology: Human Perception and Performance
30(3): 513–18.
Wachs, J., Stern, H., Edan, Y., Gillam, M., Feied, C., Smith, M., & Handler, J. (2006). A real-
time hand gesture interface for a medical image-guided system.
Proceedings of Ninth
Israeli Symposium on Computer-Aided Surgery, Medical Robotics, and Medical Imaging
.
Wang, S. B., Quattoni, A., Morency, L-P., Demirdjian, D., & Darrell, T. (2006). Hidden
conditional random fields for gesture recognition.
IEEE Computer Society Conference
on Computer Vision and Pattern Recognition
2:1521–27.
3 Gesture Interfaces
106
4
CHAPTER
Locomotion Interfaces
Mary C. Whitton, Sharif Razzaque
This chapter is about locomotion interfaces—interfaces that both enable users to
move around in real or virtual spaces and make users feel as if they are moving.
Locomotion is a special type of movement: Locomotion, as used by life scientists,
refers to the act of an organism moving itself
from one place to another
. This
includes actions such as flying, swimming, and slithering. For humans, locomo-
tion is walking, running, crawling, jumping, swimming, and so on. In Figure 4.1,
the user moves from one place to another by leaning to control speed and direc-
tion of the Segway Personal Transporter (PT); in Figure 4.2, the user is (really)
walking on the treadmill (shown in detail on the right) to move through a virtual
landscape.
The focus of this chapter is computer-based locomotion interfaces for moving
about in computer-generated scenes. One way to think of these interfaces is that
they are virtual-locomotion interfaces for virtual scenes. We may occasionally
use an example of locomotion in the real world, but when we say locomotion
interface, we mean
virtual
-locomotion interface.
Many common virtual-locomotion interfaces take input from only the user’s
hands; for instance, three-dimensional (3D) computer video games are operated
by joystick, keyboard, mouse, and/or game controller. Other interfaces use more
of the body. For example, in the real world, Segway PTs, hang gliders, and skate-
boards require users to lean and shift their weight to control motion. In a virtual-
reality system that has a real-walking locomotion interface, people use their
bodies in a completely natural fashion to walk about the virtual scene (Figure 4.3).
This chapter concentrates on virtual-locomotion interfaces that involve all of, or at
least much of, the user’s body.
Even though both arms and both legs may be involved in controlling a vehicle
and the whole body may be involved in receiving feedback about the state of the
vehicle, simulating movement resulting from operating a vehicle is conceptually
different from simulating the movement of a human walking about human-scale
spaces such as buildings and ships. In a vehicle, users’ intentions of what direction
to move and how fast to go are specified unambiguously by their interaction with
the controls of the vehicle. The challenge for virtual-locomotion interfaces is to
capture the user’s intent using data that can be derived from sensing the
pose
(position and orientation) and
movement
of the user’s body.
The scope of this chapter is, then, interfaces between human users and computer-
generated scenes that enable users to control their locomotion through the (virtual)
scene by moving their bodies. The body pose and movement specify how they want
to move—both the
direction
they want to move in and
how fast
they want to move.
FIGURE
4.1
Segway i2 self-balancing Personal Transporter.
The user controls the speed by leaning forward or backward and turns by leaning
to the right or left with the LeanSteer frame. (Courtesy of Segway Inc.)
4 Locomotion Interfaces
108
FIGURE
4.2
University of Utah’s Treadport virtual-locomotion interface.
Left:
The user walks on a treadmill while viewing the moving virtual landscape on
the large projector screens.
Right:
To simulate hills, the entire treadmill can tilt
up. To simulate the user’s virtual inertia, the Treadport physically pushes or pulls
the user via a large rod that connects to a user-worn harness. (Courtesy of
University of Utah, School of Computing;
#
John M. Hollerbach.)
FIGURE
4.3
Real-walking virtual locomotion interface.
To move about the virtual scene (
right
), the user actually walks about the real-
world laboratory (
left
). (Images courtesy of the Department of Computer Science,
UNC at Chapel Hill.)
4 Locomotion Interfaces
109
There are three dominant metaphors for virtual locomotion. In
real-
walking–style
systems, input movements and resulting movement through the
space are as natural and as much like really walking as possible. Examples are
two-dimensional (2D) treadmills, walking-in-place, and real-walking interfaces.
In
vehicle-style
interfaces, input movements and response are similar to driving
a vehicle. For example, Fleming and colleagues (2002) developed a joystick inter-
face in which pushing the joystick up (away from you) moves you forward, and
pushing the joystick left rotates you left but does not move you forward.
To move left, you must first turn (rotate) left using the joystick and then move
forward. This is similar to how a driver operates a tank.
Both real-walking and vehicle-style interfaces fall into a category of locomo-
tion techniques that Slater and Usoh (1994) call
mundane
, that is, they attempt
to mimic, as closely as possible, how we walk or control vehicles in the real world.
In contrast,
magical-style
interfaces are those that permit movements that have no
natural corollary in the real world but that serve a useful purpose when you are
moving about in a virtual scene. For instance, the ability to teleport between
two distant locations is a magical component of a locomotion interface. Similarly,
the ability to adapt the length of your virtual stride so that you move miles with
each step is a magical property.
Regardless of the choice of metaphor, the interface has to convert the user’s
physical movement (in the real world) into virtual movement—both direction
and speed—in the virtual scene. Table 4.1 gives some examples of body move-
ments and how they might be used to control locomotion direction and speed.
Note that locomotion is distinct from
wayfinding
, which refers to the cognitive
task of determining a route for reaching a destination. We avoid using
wayfinding
because it is used in the literature to refer to both wayfinding and locomotion.
Other authors have referred to movements such as going from a virtual desk to
a virtual bookshelf as
travel
(e.g., Bowman, Koller, & Hodges, 1997), but we prefer
TABLE 4.1 Examples of How Body Movement Might Be Interpreted to Control
Direction and Speed of Locomotion
Body Part Direction Control Speed Control
Hands Move in direction user is pointing Distance between hands controls step size
Feet Move in direction feet are pointing Step faster or slower to speed or slow pace
Legs Swing leg sideways from hip to
move sideways
Step faster or slower to speed or slow pace
Arms Make “banking” gesture to turn like
bird
Flap arms faster or slower to speed or
slow pace
4 Locomotion Interfaces
110
locomotion
1
because travel implies a large distance—one
travels
from New York
to Chicago, but does not travel across a room. Hollerbach’s (2002) definition of
a locomotion interface requires that the user make repetitive motions such
as tapping a key on a keyboard or walking-in-place. We do not use Hollerbach’s
definition in this chapter.
4.1
NATURE OF THE INTERFACE
The essence of a locomotion interface (for both real and virtual worlds) can
be generalized. Users communicate their intentions to the system (computer or
vehicle) by making real-world motions—they press keys, move a joystick, move
their hands, lean forward, push with a foot, and so on. These user input motions
are then converted through an interface system to movement through the real or
virtual world. The system may also provide feedback to the users, indicating how
they are moving or have moved. Users use the feedback to plan their next motion
and, for visual feedback, to help maintain balance and to update their mental
model of their surroundings.
4.1.1 The Physiological Nature of the Interface:
Perceiving Self-Motion
There are several sensory channels that provide us with information on how and
where we are moving. Each of these sensory channels—auditory, visual, vestibular
(from the inner ear), and proprioceptive (the sense of body position)—contributes
information to awareness of self-motion. Humans rely on sensory cues to deter-
mine whether they themselves are moving (self-motion) or if the objects around
them are moving (external motion) and for balance and orientation (Dichgans &
Brandt, 1977).
Auditory:
Humans have the ability to deduce qualities of their environment from
the way the environment sounds; for instance, large rooms sound different
from small rooms. We also have the ability to localize sound sources, and as
we move the perceived location of a stationary sound moves in relation to our
head. A moving sound source can, by itself, cause a stationary person to feel
as if she is moving (Lackner, 1977).
Tactile, proprioceptive, and podokinetic:
Humans can sense movement in their
joints, muscles, and viscera, and can sense pressure and slippage on the skin.
These cues are important for walking, as they tell a person where her limbs are
1
Researchers may have adopted the term
travel
because the verb form of locomotion, to
locomote
, sounds awkward.
4.1 Nature of the Interface
111