Kortum P. (ed.) HCI Beyond the GUI. Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

Подождите немного. Документ загружается.

The numbers in Table 3.3 are the static constraints for gesture postures.

However, there are also dynamic constraints—that is, intra- and interfinger

constraints, as in the following list.

FIGURE

3.13

Side view of a hand.

Finger and wrist motion direction is shown with arrows. For example, bending

fingers into a fist is flexion of joints. Stretching fingers is extension of joints.

TABLE 3.3 Range of Motion in Degrees in Hand and Wrist Joints

Joint

Extension/

Flexion Hyperextension

Adduction/

Abduction

Metacarpophalangeal (MCP)

90 0–45 30

Proximal interphalangeal (PIP) 100 0 0

Distal interphalangeal (DIP) 80 0 0

Trapezio-metacarpophalangeal (TMCP) 90 10 0

Thumb distal interphalangeal (TDIP) 80 15 0

Palmar

Adduction/

abduction

Radial

Adduction/

abduction

Thumb basal joint (TBJ) Contact/45 Contact/60

Pronation/

supination

Radial/

ulnar

Wrist

70/75

70/85 20/35

See Figure 3.12.

Rotation is done with the entire forearm.

More than one joint.

3 Gesture Interfaces

F Intrafinger constraints are the dependencies between the finger

joints.

F Interfinger constraints are the dependencies between the postures of

neighboring fingers.

Furthermore, finger postures affect pressure in the carpal tunnel (Keir et al.,

1998), most severely with metacarpophalangeal joint angles at zero degrees

(fingers extended) and at least 45 degrees.

Major principles in ergonomics (see Grant, 2002; Hedge et al., 1999; Shaw &

Hedge, 2002) follow:

F Avoid outer positions.

F Avoid repetition.

F Relax muscles.

F Relaxed neutral position is in the middle between outer positions.

F Avoid staying in a static position.

F Avoid internal and external force on joints and stopping body fluids.

Given these guidelines, a gesture interface is potentially ergonomically superior to

physical handheld devices, which introduce external force. When you consider

and compare biomechanics in gestures, it is important to evaluate internal forces

and posture angles. Such evaluation is applied in the following paragraphs to a

commonly used gesture.

The TV control gesture in Freeman & Weissman (1995) was a tight fist. Feed-

back was provided to navigate a pointer on the screen. The dialog was very sim-

ple, but tests showed that the gesture was very tiring. The ergonomics of the

gesture support this conclusion: The neutral position is for all joints to be in the

middle of respective ranges of motion. This means that a fist is a forced position

by the muscles in the hand. Furthermore, the hand is raised to head height, or

higher, which places the shoulder in or close to an outer position of external

rotation (rotated away from the body), with the weight of the arm and hand on

it. On the positive side, the wrists are kept straight while operating the TV.

The gestures in Figure 3.8 are stressing because they do not follow the inter-

finger constraints of the hand. While constraint limitations are different for each

person, the user must use more or less force to position the fingers for the system

to recognize the correct posture. Furthermore, the recognition is vulnerable to

how stretched the fingers are. This means that the users must stretch them to

outer positions.

The ergonomics show that making the recognition algorithms tolerant to non-

stressing movements is important, which allows the user to avoid remaining in a

fixed or static “residue” or “pointing” gesture. Tolerance for deviations in gestures

is desirable when implementing gesture interfaces, also because of varying hand

shapes and posture performance.

3.4 Human Factors Involved in Interface Design

3.5

DESIGN GUIDELINES

In this section we discuss some important issues in designing a gesture interface.

3.5.1 The Midas Touch

In Greek mythology, the hedonist King Midas helped Dionysius, the god of wine.

As a reward Midas asked that Dionysius make it possible for all that he, Midas,

touched to turn into gold. And so it was until he realized that everything turned

to gold, including his daughter and his food and drink. Seeing his gift as a curse,

he prayed to Dionysius and he was told to pass on the power to a river, Pactolus,

which historically has been rich in gold and other metals.

The relevance of this story to gesture interface design is important: The Midas

touch refers to the ever-returning problem of when to start and stop interpreting a

gesture. As a designer you can expect spontaneous gestures from users all the time

if the goal is natural immersive behavior. Therefore, the gesture recognition must

be very tolerant. Otherwise, users would suffer rigid constraints to their behavior

while in the system. Unfortunately, designers often select rigid constraints as solu-

tions, such as forcing the user not to move between gestures. Alternatively, users

may have a manual trigger that tells the system when a gesture starts and stops.

3.5.2 Cultural Issues

It is widely recognized that nonverbal communication is culturally dependent

(Cassell, 1998; Agliati et al., 2006) in typology (semantics), rhythm, and fre-

quency. Perhaps there are even gender differences, but this is an infant theory

in HCI (Hall, 2006). An important question to ask is whether cultural dependence

is a problem. Conventional interfaces that are international are generally in

English, but most software is available with a series of national language packages,

and people in some nations use different keyboards.

In a gesture interface, this can be translated to selectable gesture vocabularies

if it should become a problem that an emblem is illogical to another culture.

Furthermore, if a culturally dependent gesture is used, this does not necessarily

mean that it is utterly illogical for people of other cultures to learn it.

It is critical to consider cultural aspects when analyzing and developing gesture

interfaces/detectors with a focus on natural human conversation and behavior.

The system must be able to distinguish (and/or synthesize) those parameters on

rhythm, frequency, and typology.

3.5.3 Sequencing

The choice of gestures may depend on your choice of sequencing. Avoiding problems

later in the process is easier if you design the sequencing from the start. Sequence

3 Gesture Interfaces

design involves deciding each step that a user and the system will go through to

accomplish a given task. For example, one way to design an insert task follows:

1. The user makes a command gesture for insert and the gesture is

performed at the location of the object’s placement.

2. The system feeds back a list of objects to insert.

3. The user uses a pointing gesture to select the type.

4. The system removes the object list and inserts the object.

An alternative sequencing could be the following:

1. The user makes a pointing gesture at an object zone next to the workspace.

2. The system highlights the object.

3. The user uses a pointing gesture in the workspace to place the object.

4. The system inserts the object.

Note that an emblem for insert is only needed in the first example. It is obvious

that sequencing can affect interface efficiency and learnability. This is why classi-

cal WIMP interfaces usually provide multiple paths to the same objective using

the mouse only, the keyboard only, or a combination of the two. In a gesture

interface, which is quite limited, this may not be a feasible solution.

In conclusion, choose a sequencing that works well for the intended user

group and their preferred gesture vocabulary. It may be a matter of habit, which

you can determine from a Wizard of Oz experiment or by observing users’ behavior

in existing applications.

3.5.4 Use Context Awareness

As discussed in Section 3.3.1, human perception relies on context awareness and

expectations related to this context and personal attributes. Furthermore, a WIMP

interface has a context-aware menu by right-clicking on an object. This metaphor

can be implemented into your gesture interface as well. Context-dependent recogni-

tion can be used for limiting the gesture vocabularies and to solve ambiguity.

Spatial Zones (Surround GUI)

One technically tangible way of using context awareness is by using spatial zones

(Figure 3.7). The space around the user can be divided into various interaction zones

such as a workspace, menu space, template space, and command space. The position

where a gesture is performed directs the expectation of the gesture recognition.

Input Modes

Another method is to reuse gestures in different input modes. Metaphorically this

is similar to changing input modes for the mouse in the WIMP interface. For

example, the mouse in Adobe Photoshop can be used for drawing or for cutting,

depending on the input mode.

3.5 Design Guidelines

Use of “Arguments”

A third method is to assign arguments to gestures. An argument can be a physical

or a virtual placeholder that alters the meaning of a gesture.

In the project called ARTHUR (Aish et al., 2004), these arguments were phys-

ical placeholder objects made out of colored wooden bricks. Architects would use

these placeholders to move virtual buildings around in a virtual model of a city.

Other placeholders were used for parameter selection or to modify the visual

appearance of the virtual buildings. This solution provided robust selection and

placement as well as intuitive interaction for the users, who had natural tactile

and visual feedback when they moved objects (Figure 3.14).

3.5.5 Feedback

Using a hands-free interface can make the user feel disconnected from the sys-

tem. It is important to maintain a connection by supplying the user with feedback

that tells her that the system is aware of her and what she is doing. This feedback

can also tell the user in which context she is acting and how the system interprets

the actions.

The feedback can be any sensory input (tactile, auditory, or visual) depending

on the nature of the conveyed information. For example, if the user interacts with

3D objects in a workspace, the object under the hand is highlighted in a way other

than the selected objects are. This serves two goals: (1) the user knows that the

system is aware of the hand, and (2) any discrepancy between the physical world

FIGURE

3.14

ARTHUR project.

Left:

Architects investigate the location and size of a new building in London.

Right:

Physical placeholder objects are moved in place to select the number of

beams in a rooftop.

Source:

From Moeslund et al. (2004).

3 Gesture Interfaces

and the virtual environment is calibrated in the consciousness of the user. The

outcome is that the user feels freer to move less carefully in the interface.

When the user moves an object close to another object, a sound alerts the

user at the time the objects intersect while simultaneously the intersected parts

are colored red. The sound tells the user exactly when the intersection occurs

and allows the user to react promptly. The color shows the spatial extent of the

intersection.

3.5.6 Integration and Disambiguation

Designing a gestures-only interface is difficult. The taxonomy for cooperating with

other modalities is called types of cooperation (TYCOON):

Transfer:

Serial chronology, that is, first one and then the other mode.

Equivalence:

Either or, that is, use one or the other mode.

Complement:

Put that there (e.g. Say “Put” while pointing at “that” and moving the

hand to the destination and saying “there” to complete the action).

Redundancy:

Simultaneous modes to ensure robustness.

Specialization:

“Monopoly,” that is, only one mode has access to the function.

One way to organize the types of cooperation for an imaginary application

is shown in Figure 3.15. Consider having three input modes: speech, vision, and

traditional WIMP. Each function in the application should be assigned to one or

more modes using TYCOON. For example, “Exit” is a sensitive function, which

should not be accessed by accident. Thus, it is determined that all three modes

Confirm (E)

Pause (S)

Transfer

Equivalence

Complementary

Redundancy

Specialization

Select all (E)

Configuration (S)

Assign PHO(T)

Moving objects (C)

Exit (R/T)

Vision-based

Speech

Tycoon

Traditional

Scaling (S)

FIGURE

3.15

Organizing types of cooperation.

Assignment of functions to three modalities using the TYCOON taxonomy.

3.5 Design Guidelines

should be used simultaneously to exit the application. The user must say “Exit”

and make an exit gesture (redundancy), and then confirm via the keyboard

(transfer).

When using multiple modalities, there is a risk of conflicting information. An

advance expectation module can prevent such conflicts, or two modes can also

disambiguate information. Another solution to conflicting information is to assign

mode priority

and

action priority

. One mode or action can have precedence over

the other. For example, when one mode says

Yes

and the other says

, mode pri-

ority given to the first leads to system execution of the

Yes

command. If the action

is given priority, in this instance the

command is executed.

3.6

HOW TO BUILD AND TEST A GESTURE

VOCABULARY

Section 3.4 outlined the importance of choosing a logical and ergonomic gesture

vocabulary, and discussed parameters that can be tuned to achieving the same.

This section presents a procedure for how to find an appropriate gesture vocabulary

for a given application. The description is accompanied by a practical example that

shows how it can be used.

To ensure intuitive and logical mapping, an investigation into the interfaces

of known applications should be conducted to identify the needed functionalities.

It is important to remember that a gesture vocabulary must be tailored for the pur-

pose of the application and for the user group of the application. A gesture vocab-

ulary should not be imposed arbitrarily on any given application.

In the development of interfaces, scenarios have proven valuable (Dzida &

Freitag, 1998) to define the context and functionalities and to investigate the user

and problem domains. A scenario can be used to examine human–human nonverbal

communication. Users would participate in scenarios in which they communicate the

same things to a person that they would communicate to the computer application.

There are bottom-up and top-down approaches to this investigation. The bottom-

up type finds matching gestures to functions, while the top-down approach presents

gestures and identifies logically matched functions. Outlines of variations on these

approaches follow:

Subconscious, bottom-up:

Create scenarios of communication between people.

Record the transactions on video and analyze the subconscious gesturing that

occurs.

Conscious, bottom-up:

Ask for each functionality for which a gesture would be used,

or make a guided drawing test. A testee receives a random drawing project and

guides the operator with gestures on what to draw. If an object template–based

application is considered, those objects can be provided on the table.

3 Gesture Interfaces

Top-Down:

Ask each testee what a given gesture could mean, or do a drawing test.

The operator guides the testee with gesturing what to put on paper using a pre-

defined gesture vocabulary. This is useful for testing a gesture vocabulary.

Another necessary tool is a benchmark to measure the quality of a gesture

vocabulary according to principles in the human-based approach.

3.6.1 The Procedure to Find Gestures

This section describes the proposed procedure and benchmark when developing a

gesture interface. In the online case study referred to in Section 3.7, we demon-

strate our own experiment using the procedure. It is an iterative approach, where

gestures are found and tested and results are used to refine the choices.

Step A: Find the Functions

Find the functions that the gestures will have to access. Keep in mind the user interface

in existing similar applications on standard interfaces. (For example, we were design-

ing an architectural design application, so we looked at 3D Studio, among others.)

Step B: Find Logical Gestures

Find the logical gestures that represent the functions found in Step A. This can be

done through interviews or experiments with people by taking them through sce-

narios under camera surveillance where they communicate the same messages

that they would communicate to the computer to the “operator” (i.e., the person

who conducts the experiment).

It is important that the scenarios take the testees away from normal technical

thinking, especially when conducting the tests on technical people. If you tell an

engineer that it is a test for the gesture recognition system, the engineer will have

a conscious design in mind and act according to the hypothetical restrictions of

that design. In brief, you risk that such a person will still think in terms of inter-

faces and algorithms. If you want to write a scenario with a technical interface

aspect, it can be performed as a Wizard of Oz experiment, which tests not only

the gesturing but also the design of the entire interface, including the feedback

from the system and the sequencing in the interface.

The number of people required for this investigation depends on how broad

the user group is and how diverse the results of the test are. Because it is a

time-consuming process, using up to 10 participants is typical.

Step C: Process the Data

Note and capture the frames with the commonly used gestures, and note how con-

sistently the various testees use them. Note if they are used only as static postures

or if dynamics play an important part in interpreting the gesture.

3.6 How to Build and Test a Gesture Vocabulary

The theory discussed in Section 3.4.3 should be taken into account in the

selection of gestures:

F Evaluate internal force caused by posture.

– Deviation from neutral position

– Outer limits

– Forces from interjoint relations

F Evaluate frequency and duration of that gesture.

F Consider effect on wrist from wrist and finger posture.

See the online case study (Step C) to examine how this is done in praxis.

Step D: Benchmark the Chosen Gesture Vocabulary

The final step is to test the resulting gesture vocabulary. The attributes shown in

Table 3.4 are to be tested in the benchmark.

Test 1: Guess the Function Give the testee a list of functions. Present the ges-

tures and explain the general idea of the application. Ask the person to guess

the functions. Gestures that depend on context must be presented in context,

but not how the application reacts to them. Explain carefully that it is the ges-

tures that are being tested, not the intelligence of the testee. This takes tension

away from the testee.

Score ¼ wrong guesses divided by number of gestur es

Test 2: Memory This test measures how long it takes to become comfortable

with the gestures. Only when the gestures are performed without hesitation will

the user keep the application in focus rather than the interface.

Give the gesture vocabulary to the testee, who will experiment with the

gestures to make sure they are understood. Remind the testee again that it is the ges-

ture vocabulary that is subject to the test, not the testee’s ability to perform it.

TABLE 3.4 Overview of What is Actually Tested in Each Test

Attribute Evaluated in Test ___

Semantic interpretation 1

Generalization 1

Intuitivity 1, 2

Memory, learning rate 2

Stress 3

3 Gesture Interfaces

100

Present a slide show of functions at a swift pace, two seconds per function. The

testee must perform them correctly. Continue until they are all correct. The order

should be logical toward sequences in the application, but it should not be the same

every time. Then the testee will simply memorize the order of gestures and not the

mapping. Restart the slide show after every mistake, and show the gesture vocab-

ulary to the testee between each retry.

Score ¼ number of restarts

The outcome is an indicator of how difficult it will be for a new user to become

comfortable with the gesture vocabulary to the extent that the gestures will not

steal the focus from the user’s tasks.

Test 3: Stress This is a subjective evaluation of ergonomics. Present the testee

with a list of a sequence of gestures. The testee must perform the sequence

times,

where

times the size of the gesture vocabulary equals at least 200. Go back to the

neutral hand position (residue) between each gesture. Afterward ask the testee how

stressful each gesture was and to provide an overall rating for using each gesture

over a long time. Meanwhile the operator should note how ergonomically the testee

is performing the gestures and how the testee is sitting at the table.

We used the following score list for each gesture and overall for the

sequence:

1. No problem

2. Mildly tiring/stressing

3. Tiring/stressing

4. Very annoying/painful

5. Impossible

To place the process in context, as it is very subjective, a computer mouse

would be scored at around 2 for normally able people. A more objective stress test

would require electrodes on the nerves at the wrist and forearm to measure strain.

Such testing may be of interest to pain researchers in particular.

The benchmark can be used to compare multiple gesture vocabularies or user

groups. Test 2 is only comparable if the vocabularies are of the same size. If test-

ing a single vocabulary, reasonable success criteria must be ascertained. These

aims depend on the gesture vocabulary at hand. See the online case study (Step D)

to see how this is done in praxis.

Wizard of Oz experiments have proven valuable in the development of ges-

tures (Beringer, 2002; Carbini et al., 2006). The experiments simulate the response

of the system by having a person respond to the user commands. This approach

tests a developed interface and can identify problems, with the choice of gestures

and sequencing as the necessary feedback.

3.6 How to Build and Test a Gesture Vocabulary

101