Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

Подождите немного. Документ загружается.

Tan, H. Z., Durlach, N. I., et al. (1994). Tactual performance with motional stimulation of

the index finger.

Journal of the Acoustical Society of America

95(5):2986–87.

Tan, H. Z., Durlach, N. I., et al. (1993). Manual resolution of compliance when work and force

cues are minimized.

Proceedings of ASME Dynamic Systems and Control Division

, 99–104.

Tan, H. Z., Eberman, B., et al. (1994). Human factors for the design of force-reflecting hap-

tic interfaces.

Proceedings of International Mechanical Engineers Congress and Exposi-

tion

, Chicago, 353–59.

Tan, H. Z., Pang, X. D., et al. (1992). Manual resolution of length, force, and compliance.

Proceedings of American Society of Mechanical Engineers

, Winter Annual Meeting, Ana-

heim, CA, 13–18.

Tan, H. Z., & Pentland, A. (1997). Tactual displays for wearable computing.

Digest of the

First International Symposium on Wearable Computers

, 84–89.

Tang,H.,&Beebe,D.J.(2006).Anoraltactileinterfaceforblindnavigation.

IEEE

Transactions on Neural Systems and Rehabilitation Engineering

14(1):116–23.

Traylor, R., & Tan, H. Z. (2002). Development of a wearable haptic display for situation

awareness in altered-gravity environment: Some initial findings.

Proceedings of Tenth

Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems

159–64.

Tsagarakis, N., Caldwell, D. G., et al. (1999). A 7 DOF pneumatic muscle actuator (pMA)

powered exoskeleton.

Proceedings of Eighth IEEE International Workshop on Robot and

Human Interaction

, 327–33.

Tsagarakis, N. G., & Caldwell, D. G. (2003). Development and control of a “soft-actuated”

exoskeleton for use in physiotherapy and training.

Autonomous Robots

15(1):21–33.

Unger, B. J., Nicolaidis, A., et al. (2001). Comparison of 3-D haptic peg-in-hole tasks in real

and virtual environments.

Proceedings of IEEE/RSJ International Conference on Intelli-

gent Robots and Systems

, 1751–56.

Upperman, G., Suzuki, A., et al. (2004). Comparison of human haptic size discrimination

performance in simulated environments with varying levels of force and stiffness.

Pro-

ceedings of International Symposium on Haptic Interfaces for Virtual Environment and

Teleoperator Systems

, 169–75.

Verillo, R. T. (1966). Vibrotactile sensitivity and the frequency response of the Pacinian

corpuscle.

Psychonomic Science

4:135–36.

Weisenberger, J. M., Krier, M. J., et al. (2000). Judging the orientation of sinusoidal and square

wave virtual gratings presented via 2-DOF and 3-DOF haptic interfaces.

Haptics-e

1(4).

West, A. M., & Cutkosky, M. R. (1997). Detection of real and virtual fine surface features

with a haptic interface and stylus.

Proceedings of ASME International Mechanical Engi-

neers Congress Dynamic Systems and Control Division

, Dallas, 159–65.

Wiker, S., Hershkowitz, E., et al. (1989). Teleoperator comfort and psychometric stability:

Criteria for limiting master-controller forces of operation and feedback during telemanipu-

lation.

Proceedings of NASA Conference on Space Telerobotics

, Greenbelt, MD, 99–107.

Williams, L. E. P., Loftin, R. B., et al. (2002). Kinesthetic and visual force display for telero-

botics.

Proceedings of IEEE International Conference on Robotics and Automation

1249–54.

Williams II, R. L., Murphy, M. A., et al. (1998). Kinesthetic force/moment feedback via

active exoskeleton.

Proceedings of Image Society Conference

2 Haptic Interfaces

Wood, J., Magennis, M., et al. (2003). The design and evaluation of a computer game for the

blind in the GRAB haptic audio virtual environment.

Proceedings of Eurohaptics Conference

Yu, W., Ramloll, R., et al. (2000). Haptic graphs for blind computer users.

Lecture Notes in

Computer Science

2058:41–51.

Zanganeh, K. E., & Angeles, J. (1997). Kinematic isotropy and the optimum design of

parallel manipulators.

International Journal of Robotics Research

16(2):185–97.

Zerkus, M., Becker, B., et al. (1993). Temperature sensing in virtual reality and telerobotics.

Virtual Reality Systems

1(2):88–90.

Zhu, H. (1988). Electrotactile stimulation. In Webster, J., ed.,

Tactile Sensors for Robotics

and Medicine

. New York: John Wiley & Sons, 341–53.

References

This page intentionally left blank

CHAPTER

Gesture Interfaces

Michael Nielsen, Thomas B. Moeslund,

Moritz Sto

rring, Erik Granum

This chapter provides an introduction to the domain of gesture interfaces. We

begin with the foundation of gesture interfaces and how and where they are com-

monly used, followed by basic theory that will help you select and design a gesture

interface, and related advice and warnings. Next, we present a procedure to iden-

tify and test good gestures for a design and a practical example of how to use this

procedure. Finally, we provide a short perspective on our vision of the future as

the interface evolves.

3.1

GESTURES

Gestures originate from natural interaction between people. They consist of

movements of the body and face as nonverbal communication that complements

verbal communication. This is the inspiration behind using gesture interfaces

between man and machine. Figure 3.1 shows an interaction between two people

that relies on nonverbal communication. The camera is mounted on the ear of

one person. As seen here, interpretation is context dependent. They cannot hear

each other because of loud music, and the person with the camera wants to order

a drink. The gesture by the person at the left means “I cannot hear you.” The per-

son (at the right) with the camera on his head is using an iconic gesture to show

what he wants. The other person imitates his gesture.

A gesture interface can be seen as an alternative or complement to existing

interface techniques, such as the old desktop paradigm. Good examples are the

new alternatives to the mouse, such as ergonomic track balls, mouse tablet pens

(e.g., from Palm and a TabletPC), and the iGesturePad (Figure 3.2).

Gesture interfaces can navigate a Windows interface just as well or better than

the mouse cursor, while they may be more or less useless when it comes to fast

computer games, such as three-dimensional (3D) shooters and airplane simula-

tors. For these applications, specialized interfaces have been developed that are

based on wheels and flight sticks from the real world.

When developing a gesture interface, the objective should not be “to make a

gesture interface.” A gesture interface is not universally the best interface for

any particular application. The objective is “to develop a more efficient interface”

to a given application.

This can be illustrated by an example of an interface for artistic modeling of a

sculpture. The artist may be given a mouse and a keyboard for a CAD program.

The result is perfect to the smallest detail regarding accuracy of the lines, because

Mouse4

Mouse3

Mouse5

Wheel

down

iGesture Pad

FIGURE

3.2

The iGesturePad reacts to the relative positions and movements of the user’s

fingertips as they touch the pad. (Courtesy of Fingerworks.)

FIGURE

3.1

Interaction based on nonverbal communication.

Gestures between two people who cannot hear each other.

3 Gesture Interfaces

it is possible to set coordinates explicitly. If the artist is provided with a gesture

interface in which a virtual clay model can be altered by touching and squeezing

it, it will not be accurate in terms of coordinates and straight lines, but it might

be aesthetically closer to the artist’s vision. Thus, selecting an interface is a matter

of which outcome of the application is desired.

Consequently, the first step is analysis of the kind of interface that is most

suitable for this task. Such analysis may lead to the conclusion that a gesture inter-

face is the most suitable type of interface.

3.2

TECHNOLOGY AND APPLICABILITY

Much work has been done in the investigation and development of natural interac-

tion interfaces, including gesture interfaces (Cassell, 1998; Freemann & Weissman,

1995; Hummels & Stapiers, 1998; Storring et al., 2001; Paggio & Music, 2000;

Steininger et al., 2002; Streitz et al., 2001). Gesture interfaces have also been devel-

oped in science fiction literature and movies, such as the movies

Johnny Mnemonic

(1995),

Final Fantasy

(2001), and

Minority Report

(2002). Furthermore, gesture

interfaces are applied to solve problems for people with physical disabilities (Keates

& Robinson, 1998; Jaimes & Sebe, 2005).

The most interesting potential in this field of research is to make accessory-

free and wireless gesture interfaces, such as in virtual-reality and intelligent

rooms, because the use of physical and wired gadgets makes the interface and ges-

turing tedious and less natural. The first solutions required expensive data gloves

or other such intrusive equipment with wires that made the user feel uncomfort-

able. Greater success came with pen-based gestures (e.g., Palm handheld devices),

where trajectories were recognized as gestures.

A common motivation behind the analogy between nonverbal communica-

tion and human–computer communication is that it allows for better, more natu-

ral and intuitive interaction. However, Cassell (1998) claims that this cannot be

liberally assumed to always be the case. For example, a command language is as

natural as writing a to-do list. Another idea involved accessibility studies, where

people with physical disabilities are not capable of using tactile input modalities.

While natural gestures are often subtle, gesture interfaces rely on emphasized

gestures. Interfaces have mostly revolved around simple hand gestures for point-

ing out objects or controlling a mouse-like pointer and a few gestures that are

linked to specific functions in the application. Simple usage of these motion-tracking

interfaces is found in game consoles (e.g., Playstation Eye Toy).

Recent work has also focused on facial gestures (face expressions and

poses)—detecting reactions and emotions. This information can be used for

automatic annotation in human behavior studies (Emotion Tool from iMotions,

http://www.imotions.dk

), accessibility for paralyzed people (Betke et al., 2002),

and feedback to an intelligent learning system (Pantic & Patras, 2006).

3.2 Technology and Applicability

3.2.1 Mechanical and Tactile Interfaces

Early gesture interfaces relied on mechanical or magnetic input devices. Examples

include the data glove (Figure 3.3), the body suit (Suguru Goto, 2006), and Nintendo Wii.

Single-point touch interfaces are well known as pen gestures (Long et al.,

2000), most commonly seen in Palm handheld devices. But recent research has

developed multipoint touches directly onto the screen, used in the iGesturePad

(Jefferson Han, 2005; Tse et al., 2006), which open up a new and more efficient

interface potential.

There are examples of research in making the computer aware of human emo-

tions shown in body language. De Silva et al. (2006) detected emotion intensity from

gestures using sensors that read galvanic skin response. However, people in general

do not like the idea of having such sensors or attachments on their bodies.

3.2.2 Computer Vision Interfaces

When the aim is to make gesture interfaces invisible to the user (Jaimes & Sebe,

2005), computer vision is a nice way to detect gestures. Computer vision is inher-

ently wireless, and people have become accustomed to surveillance cameras in

retail and airport environments.

Computer vision algorithms often consist of three parts:

segmentation

that

spots relevant parts in the field of view,

tracking

that follows the movements,

and

classification

that finds meaningful information.

FIGURE

3.3

Data glove from 5DT. (Courtesy of

www.5DT.com

3 Gesture Interfaces

Segmentation detects the body parts in the images, such as relying on shape

or skin color, or motion detection in video sequences for dynamic gestures (Stoer-

ring et al., 2004). In order to move past this step, many solutions have simplified

the segmentation by using infrared reflectors or other such markings that are

easy to segment and track.

Tracking follows a set of features when it moves with, for example, condensa-

tion or Kalman filtering (Rasmussen et al., 2006).

Classification detects the actual gestures using hidden Markov models (Sage

et al., 2003; De Silva et al., 2006; Min et al., 1999), template or model matching (Moe-

slund et al., 2003; Riviera & Guitton, 2004; Fihl et al., 2006), or fuzzy or Bayesian logic

(Aveles-Arriga et al., 2006; Wachs et al., 2006; Moustakas et al., 2006).

A new approach (Wang et al., 2006) tracks the movement of a camera phone by

analyzing its images of surroundings. These phone movements are used as gestures.

3.2.3 Face Gaze and Expression

Face gaze and expression are a subdomain of gesture research (Jaimes &

Sebe, 2005). Face gaze tracking is traditionally used for viewpoint estimation of

virtual-reality and stereo displays, but recent research aims to extract all visual

information from the face.

Facial expressions may be used as input modalities in accessibility research

(Betke et al., 2002), such as for disabled people who cannot make any movement

other than facial, such as blinking or smiling. Furthermore, facial movements con-

vey information about the emotions of the user (Pantic & Rothkrantz, 2003).

These can be used to detect confusion, user reactions, and intentions, and for pur-

poses of surveillance and automatic video feed annotation such as in human

behavior research. A computer without empathy is much like an apathetic person,

and interacting with it can cause frustration (Klein et al., 2002).

Computer vision is also used for detection of facial expressions, which allows

for an impressive perceptive interface and efficient video annotation tools, and

can give emotional awareness to virtual agents (Betke et al., 2002; Bowyer et al.,

2006; Pantic & Patras, 2006). However, the technology for doing this is still under

development and results are usually given as technical detection rates. Testing

user experience with such systems is going to be very interesting.

3.2.4 Applicability

Gesture interfaces are popular wherever the interface requires some freedom of

movement or immersive feeling such as in virtual-reality environments (Stoerring

et al., 2001), intelligent rooms (Streitz et al., 2001), medical tools during surgery

(Wachs et al., 2006), and medical simulators (Tsagarakis et al., 2006). Other typical

applications are advanced electronic whiteboards and handling of 3D objects

(Mu

ller-Tomfelde & Steiner, 2001; Moustakas et al., 2006).

3.2 Technology and Applicability

An example of a gesture interface system (Moustakas et al., 2006) is shown in

Figure 3.4. The user can retrieve 3D objects from a database and manipulate

them. Commands such as retrieve, rotate, and so on, are accessed through speech.

The user manipulates the 3D space with gestures such as showing how to rotate,

scale, move, and draw primitives. A virtual agent acts as a help wizard if the user

forgets the commands and gestures.

Gesture interface systems are also very applicable in accessibility research

because they are naturally hands-off and hands-free interactions that can be done

with any functional part of the body. Interfaces for accessibility focus on creating

alternative input modes to WIMP (windows, icons, mouse, pointers) applications,

such as controlling the mouse pointer with the face (Figure 3.5) or any single limb

that the disabled person can use (Betke et al., 2002).

3.3

FUNDAMENTAL NATURE OF THE INTERFACE

The simplest gesture interface is the well-known motion detector that turns on a

light in a room or triggers the water in a water fountain at a public lavatory. We will

focus on the more explicit gesture interfaces in which specific gestures such as pos-

tures or movements are to be interpreted by a computer to motivate something in an

application. This section describes general theory related to such gesture interfaces.

Interaction Processing

Microphone

Virtual

guide

Stereo camera

Large display

Speech

receiver

TCP

Talking-

head virtual

guide

Gesture–

speech

recognition

Virtual assembly,

search platform,

sketch recognition

3D model

database

Target

virtual

environment

Speech

trans-

mitter

FIGURE

3.4

Example of a gesture interface system.

Object search using speech and gestures and a virtual agent in Masterpiece.

Source:

Adapted from Moustakas et al. (2006);

2006 IEEE.

3 Gesture Interfaces

3.3.1 How Gestures Relate to Other Modes

It is not always feasible to design an interface that relies completely on gestures

because there may be issues relating to user or computer system precision and accu-

racy or the complexity of the input required. Hence, it is often beneficial to make

gestures part of a multimodal system. To know when to use one mode or the other,

we need to be aware of perceptive abilities in relation to other senses. Table 3.1

shows the senses that contain information to be interpreted in human perception.

Audible, visual, and tactile senses are the most important in human–computer

interaction, and these are the commonly used senses thus far to make use of

in interface design. Scent and taste are less relevant in a computer interaction

context, but they may become more important in the future as technology for

detection and synthesis emerges.

The senses can obtain conflicting, complementary, irrelevant, inadequate, or

redundant information. These data must be filtered in the cognitive system and

processed in order to be understood (Figure 3.6). This is done using expectations

based on a known or assumed context regarding the world, history, and personal

attributes. Sometimes these expectations are wrong, which leads to prejudice,

among other things. Figure 3.6 also lists what those expectations could consist of

if a similar interpretation system were developed on a computer.

If the computer would be able to interpret the human behavior and provide

natural sensory feedback, it would enhance the immersive feeling of the user.

FIGURE

3.5

Controlling a mouse pointer with the face.

The camera mouse is controlled by a 30-month-old user.

Source:

From Betke et al.

(2002);

2002 IEEE.

3.3 Fundamental Nature of the Interface