Bunt H., Beun R.-J., Borghuis T. (eds.) Multimodal Human-Computer Communication. Systems, Techniques, and Experiments

Подождите немного. Документ загружается.

Developing Multimodal Interfaces 181

structure where each branch codes for a frame slot. Instantiating a branch of this

structure brings mechanisms into play which make the memory units involved

temporarily and together sensitive to a new precise time location. This is the

way variables participating in the same relation can be bound. Thanks to the

absence of interaction between pulse Signals tuned to different time slices (so-

called 'phases'), several searches can be carried out in parallel when the system

deals with an utterance to be analyzed. Given that interpretation depends on

the simultaneous activation of several instantiated frames, and that word repre-

sentations can share the same frame, synchrony coding binds items participating

in a frame instance without interferences with other instances of the same frame.

Fig. 16. Internal representation of frame instances in 3 GPN modules : A, whose

output are word detectors; B, made of class detectors, and C which contains the frame

pathways. The long-term storage of a frame is performed through the creation of a

new pathway, whereas the short-term storage of frame instances makes use of the time

coding shown at the left. Three frame instances have been temporarily memorized

using three pulses, respectively tuned to phase 1, 2 and 3. The question

"Is there a

train to Eindhoven?"

partly activates all the frame pathways. The pathway activated

the most is facilitated at phase 3 of

"Eindhoven",

which results in the sole activation

of the

"A train"

instance of the

"train"

class.

The third point, that is the possibility of using synchrony coding while dealing

with partial data, is crucial for implementing semantic inferences if we assimilate

an input utterance with a partial version of the frames it refers to. The interpre-

tation process can be viewed as the shift of the internal activity towards other

items than the ones that are directly under the influence of the input, thanks to

'semantic' pathways. These pathways are aimed at conveying activation towards

the internal representations of relevant symbols that are not actually present

in the input, although related to the input. The actual direction taken by the

182 J.C. Martin, R. Veldman, and D. B~roule

activation depends on the way the current combination of words in the input

sentence will be able to feed into frame-like representations. The most active

frame propagates its activation towards missing input items, thus carrying out

inferences by shifting the internal activation. The principle of phase coding has

already been used in (Shastri and Ajjanagadde, 1993) to make several inferences

in parallel. In this view, compositionality relies on the partial coincidence of

input items with internal references (frames), leading to the activation of other

concerned internal items as a manifestation of the interpretation process.

5 Conclusion and Future Research

In this paper, we have presented three parts of our work concerning the devel-

opment of multimodal interfaces. First, we have described a theoretical frame-

work built around types and goals of cooperation between modalities, and we

considered examples of multimodal interaction and experimental results with

this framework. Second, we have described a command language for specifying

cooperation between modalities and illustrated it with COMIT, a multimodal

interface developed for graphical interface design. Third, we have explained the

functioning of a multimodal module made of Guided Propagation Networks.

Yhture research will include the evaluation of COMIT with several users, the

development of an interface for the itinary description application (Martin 97),

and the empowering of the multimodal module with linguistic and semantic ca-

pabilities using well-established approaches. Guided Propagation Networks and

synchrony coding are also planned to be used for reasoning purposes, a possi-

bility addressed in the last part of this paper. A multimodal application would

then take advantage of the simultaneous, adaptive and robust management of

variables within a multi-level homogeneous system.

Acknowledgments The research by Jean-Claude Martin was financed by a

DRET-CNRS grant.

References

Andr6, E. and Rist, T. (1995) Generating coherent presentations employing textual

and visual material.

Artificial Intelligence Review 9 (2-3),

147-165.

Baekgaard, A. (1995) Constraining of input media in a spoken dialog system. In

Proc. 4th European Conference on Speech Communication and Technology

(EU-

ROSPEECH '95), 1181-1184.

Bellalem, N. and Romary, L. (1995) Reference interpretation in a multimodal environ-

ment combining speech and gesture. In: Lee, J. (1995)

Pre-Proceedings First Int.

Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research

and Applications,

University of Edinburgh.

B6roule, D. (1985)

Un model~ de mdmoire adaptative, dynamique et associative pour

le traitement automatique de la parole.

Thesis, University of Paris XI, Orsay.

B6roule, D. (1988) The never-ending learning. In R. Eckmiller and C. v.d. Malsburg

(eds.),

Neural Computers.

NATO ASI Series F, vol 41. Berlin: Springer, 219-230.

Developing Multimodal Interfaces 183

B~roule, D. (1990) Guided propagation: current state of theory and application. In F.

Fogelman Souli~ and J. H~rault (eds.)

Neurocomputing,

NATO ASI Series, Vol. F

68, 241-260. Berlin: Springer.

B~roule, D., Von Hoe, R. and Ruellan, H. (1994) A Guided Propagation Model of

Reading.

Annual Progress Report 28,

Instituut voor Perceptie Onderzoek IPO,

Eindhoven, 21-29.

Blanchet, P. (1992)

Une architecture connexionniste pour l'apprentissage par

l'expdrience et la reprdsentation des connaissances.

Thesis, University of Paris XI,

Orsay.

Bolt, R.A. (1980) 'Put - That - There': Voice and Gesture at The Graphics Interface.

Computer Graphics lJ (3),

262-270.

Bos, E. (1993) Easier said or done? Studies in multimodal human-computer interaction.

NICI technical report 93-02,

University of Nijmegen.

Bourdot, P., Krus, M., Gherbi, R. (1995) Management of non-standard devices for

multimodal user interfaces under UNIX/Xll.

This volume.

Bressolle, M.C, Pavard, B., Leroux, M. (1997) The role of multimodal communication

in cooperation and intention recognition: the case of air traffic control.

This volume.

Briffault, X. (1996) Une interface multimodale pour l'aide a la navigation. Working

paper, LIMSI, Orsay. http://www, limsi, fr/Individu/xavier/index, html

Bunt, H., Beun, R. J., and Borghuis, T. (eds.)

Proceedings of the International Confer-

ence on Cooperative Multimodal Communication CMC/95.

Eindhoven, May 24-26.

Carbonnel, J.R. (1970) Mixed-Initiative Man-Computer Dialogues. Bolt, Beranek and

Newman (BBN) Report N 1971, Cambridge, MA.

Catinis, L., Caelen, J. (1995) Analyse du comportement multimodal de l'usager humain

dans une tache de dessin.

Acres des 7. Journdes sur l'Ingdnidrie de l'Interaction

Homme-Machine (IHM'95),

123-129.

Cheyer, A. and Julia, L. (1995) Multimodal maps: an agent- based approach.

This

volume.

Coutaz, J., Salber, D., Carraux, E. and Portolan, N. (1996) NEIMO, a multiworkstation

usability lab for observing and analyzing multimodal interaction. To appear in

CHI'96 Conference Proceedings Companion.

Video.

Coutaz, J. and Nigay, L. (1994) Les propri~t~s CARE dans les interfaces multi-

modales.

Acres des 6~mes Journdes sur l'Ingdnidrie de l'Interaction Homme-

Machine (IHM'94),

Lille, p. 7-14.

Escande, P., B~roule, D. and Blanchet, P. (1991) Speech recognition experiments with

Guided Propagation.

Proc. of IJCNN'91.

Daniel, M.P., Carite, L. and Denis, M. (1994) Modes of linearization in the description

of spatial configurations. In Portugali, J. (ed.),

The construction of cognitive maps.

Dordrecht: Kluwer, 297-318.

Dowell, J., Shmueli, Y., and Salter, I. (1995) Applying a cognitive model of the user

to the design of a multimodal speech interface. In: Lee, J. (1995)

Pre-Proceedings

First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces:

Research and Applications,

University of Edinburgh.

Faure, C. and Julia, L. (1994) An agent-based architecture for a multimodal interface.

Working notes of the AAAI symposium on Intelligent Multi-Media Multi-Modal

Systems.

March 21-23, Stanford.

Foote, J.T., Brown, M.G., Jones, G.J.F., Sparck Jones, K., and Young, S.J. (1995)

Video mail retrieval by voice: towards intelligent retrieval and browsing of multi-

media documents. In: Lee, J. (1995)

Pre-Proceedings First Int. Worskhop on In-

telligence and Multimodality in Multimedia Interfaces: Research and Applications,

University of Edinburgh,

184 J.C. Martin, R. Veldman, and D. B~roule

Frohlich, D.M. (1991) The design space of interfaces. In L. Kjelldahl (ed.) Multimedia:

principles, systems and applications. Berlin: Springer.

Gonqalves, M.R. (1996) Working notes on itinerary descriptions. LIMSI, Orsay.

http://www, limsi, fr/Individu/goncalve/index, html

Hare, M., Doubleday, A., Bennett, I., and Ryan, M. (1995) Intelligent presentation of

information retrieved from heterogeneous multimedia databases. In: Lee, J. (1995)

Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multi-

media Interfaces: Research and Applications, University of Edinburgh,

Han, Y. and Zukerman, I. (1997) A cooperative approach for multimodal presentation

planning. This volume.

Huls, C. and Bos, E. (1997) Studies into full integration of language and action. This

volume.

Hurault-Plantet and Briffault (1996) Atelier de g~nie linguistique et visualisation

graphique,

http://www.limsi.fr/Individu/gs/GroupeLC/Outils.html

Hutchins, E.L., Holland, J.D. and Norman, D.A. (1986) Direct manipulation inter-

faces. In Norman, D.A. and Draper, S.W. (eds.), User centred system design: new

perspectives on human computer design. Hillsdale, N J: Lawrence Erlbaum.

Inder, R., Oberlander, J., and Tobin, R. (1995) Intelligent support for navigation in

hypermedia: discourse structure and the Web. In: Lee, J. (1995) Pre-Proceedings

First Int. Worskhop on Intelligence and Multimodality in Multimedia Interface:

Research and Applications, University of Edinburgh.

Jackendoff, R. (1987) On beyond zebra: the relation between linguistic and visual

information. Cognition 26 (2), 89-114.

Lee, J. (ed.) (1995) Pre-Proceedings First International Workshop on Intelligence and

Multimodality in Multimedia Interfaces: Research and Applications. University of

Edinburgh.

Mackinlay, J., Card, S.K. & Robertson, G.G. (1990) A Semantic Analysis of the Design

Space of Input Devices. Human-Computer Interaction. voh 5, no 2-3, pp. 145-190.

Martin, J.C. (1995) Coopdrations entre modalitds et liage par synchronie dans les

interfaces multimodales. Ph.D. Thesis, TELECOM Paris.

http://www, limsi, fr/Individu, martin

Martin, J.C. (1996) Types et buts de cooperation entre modalit~s dans les interfaces

multimodales. Techniques et Science Inf ormatiques 15, 10/1996, 1367-1397.

Martin, J.C. (1997) Towards intelligent cooperation between modalities. The

example of a system enabling multimodal interaction with a map. Proc.

IJCAI'97 International Workshop on Intelligent Multimodal Systems, 63-69.

http ://www. limsi, fr: 80/Individu/mart in/ij cai/article, html

Martin, J.C. and B~roule, D. (1993) Types et buts de cooperations entre modalit~s. In

Proc. 5 th Conf. on Human-Computer Interaction IHM'93, 17-22.

Martin, J.C. and B~roule, D. (1995) Temporal codes within a typology of cooperation

between modalities. Artificial Intelligence Review 9, 1-8.

Maybury, M. (1991) Introduction. Intelligent multimedia interfaces. Cambridge, MA:

AAAI Press.

Nigay, L. and Coutaz, J. (1993) A design space for multimodal systems: concurrent

processing and data fusion. Proc. of Interchi'93, 172-178.

Nigay, L. and Coutaz, J. (1995) Multifeature systems: from HCI properties to software

design. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and

Multimodality in Multimedia Interfaces: Research and Applications, University of

Edinburgh.

O'Nuallain, S. and Smith, A.G. (1994) An investigation into the common semantics of

language and vision. Artificial Intelligence Review 8 (2-3), 113-122.

Developing Multimodal Interfaces 185

Olivier, P. and Tsujii, J.I. (1994) Quantitative perceptual representation of preposi-

tional semantics. Artificial Intelligence Review 8 (2-3).

Roques, M. (1994) Dynamic Grammatical Representations in Guided Propagation Net-

works. In R.C. Carrasco and J. Oncina (eds.) Grammatical Inference and Applica-

tions, Lecture Notes in Artificial Intelligence 862, 189-202. Berlin: Springer.

Salisbury, M.W., Hendrickson, J.H., Lammers, T.L., Fu, C., and Moody, S. A. (1990)

Talk and draw: bundling speech and graphics. IEEE Computer 23 (8), 59-65.

Santana, S. and Pineda, L.A. (1995) Producing coordinated natural language and

graphical explanations in the context of a geometric problem-solving task. In: Lee,

J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality

in Multimedia Interfaces: Research and Applications, University of Edinburgh.

Shastri, L. and Ajjanagadde, V. (1993) From simple associations to systematic reason-

ing: a connectionist representation of rules, variables and dynamic bindings using

temporal synchrony. Behavioural and Brain Sciences, 16, 417-494.

Sims, R. and Hedberg, J. (1995) Dimensions of learner control: a reappraisal of interac-

tive multimedia instruction. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop

on Intelligence and Multimodality in Multimedia Interfaces: Research and Applica-

tions, University of Edinburgh.

Siroux, J., Guyomard, M., Multon, F., and Remondeau, C. (1997) Modeling and pro-

cessing of the oral and tactile activities in the Georal tactile system. This volume.

Sowa, J. (1983) Conceptual Structures: Information Processing in Mind and Machine.

Reading, MA: Addison-Wesley.

Stern, R.M. (1995) Robust speech recognition. Section 14 in electronic book: Survey

of the State of the Art in Human Language Technology.

http ://www. cse. ogi. edu/CSLU/HLTsurvey/chlnode6, html/

Vaananen, K. (1995) Four pillars for improving the quality of multimedia applications.

In Proc. First Int. Workshop on Evaluation Methods and Quality Criteria for Mul-

timedia Applications, San Francisco.

Vo, M. T. and Waibel, A. (1993) Multimodal Human-Computer Interaction. In Proc.

International Symposium on Spoken Dialogue: New Directions in Human and Man-

Machine Communication, Tokyo, 95- 101.

Veldman, R. (1995) Experiments on robust parsing in a multimodal Guided Propaga-

tion Network. LIMSI (ERASMUS) Report 95-11, Orsay,

Wahlster, W., Andre, E., Finkler, W., Profitlich, H.J., and Rist, T. (1991) Plan-based

integration of natural language and graphics generation. AI Journal 63, 387-427.

Wang, E., Shahnvaz, H., Hedman, L., Papadopoulos, K., and Watkinson, N. (1993) A

usability evaluation of text and speech redundant help messages on a reader inter-

face. In G. Salvendy & M. Smith (eds.), Human-Computer Interaction: Software

and Hardware Interfaces, 724-729.

Westerlund, P., B~roule, D. and Roques, M. (1994) Experiments of robust parsing

using a Guided Propagation Network. In Proc. International Conference on New

Methods in Language Processing (NEMLAP'94) Manchester.

Webber, B. (1997) Instructing Animated Agents: Viewing Language in Behavioural

Terms. This volume.

Yankelovich, N., Levow, G., Marx, M. (1995) Designing Speech Acts: Issues in Speech

User Interfaces. Proc. of CHI '95, Conference on Human Factors in Computing

Systems.

186 J.C. Martin, R. Veldman, and D. B4roule

Appendix: Computation of Guided Propagation Network

Parameters to Obtain a Multimodal Recognition Score

The following parameters of a CD unit participate in multimodal recognition scores:

- the Ratio (R~) between context and stimuli contributions (R~ is the contextual

weight if the Stimulation weight is equal to 1),

- the Excitability (Ei), which defines the cell threshold in the following way:

Amax *

(1 -}- Ri-1)

where A,~, is the maximum value of the signal delivered by a cell (Fig. 17).

RI. 1

Output A

C | T

Transfer function

Input

+K,* ci.l

Fig. 17. Parameters of a CD unit. Each CD unit computes an output value C~ as a

function of its incoming contextual signal C~-I and stimuli S~ (to the left). To obtain a

recognition score which is a function of the differences between the detected sequence

of events and the prototype of a command, the transfer function has to be linear (to

the right-hand side).

The output value should be equal to the maximum Amax for the maximum input:

(S, + C~-l)m~x = A,~x * (1 + R~-I)

Developing Multimodal Interfaces 187

In the simplified case, 0i : 0 and the cell output is given by:

& + R~-I * C~-I

C,=

1 q- P~a-1

By replacing 0~-1 by its value in the above recursive formula, we obtain the fol-

lowing expression, in term of a sum of weighted stimuli.

C~-m,&

1 +

Rn-1

R,~- I

"+ (1 + R,-1)O + R,-2) * S,-1 (1)

Rn--1 * Rn-2

+ *S~-2

(1 + R~_l)(a + R~-~)0 + R~-3)

Rn-1 * Rn-2... *

+ * &

(1 + R,~-I)O + Rn_~)... (1 + Ro)

If C~ is the output of a pathway comprising n cells and which is therefore fed by

n stimuli, it should follow the conditions below :

Vi C [1,n],&= Am,~ ~ C,~ = Ama,

in other words, if the pathway is fed by all its stimuli with a maximum value, its

output should be maximum. This can be verified by replacing every & by

Ama,

in (1),

provided that Ro = 0.

C,~ = A,~,~/p

lip

of the stimuli is actually participating to the pathway acti-

vation. This should be true for the limit case: when only one stimulus activates the

pathway, whatever its location in time and along the path.

Am~x

C~-

* Amax

1 + Rn-1

R,~- I

9 Amax

(1 + 1~,-i)0 + R,~_~)

Rn-1 * Rn-2

(1 + R._l)(t + R.-~)O + R._~)

* Amax

(2)

(a)

Rn-1 * Rn-2 * ... * R1

9 Amax

(1 + Rn-1)O + R,_~)... (1 + Ro)

The first equality (2) gives R~-I = n - 1, from which (3) gives R~-2 ---- n - 2, and

so one until R0 -- 0.

Cooperation between Reactive 3D Objects and a

Multimodal X Window Kernel for CAD

Patrick Bourdot, Mike Krus, Rachid Gherbi

LIMSI-CNRS UPR 3251, University Paris XI, BP 133, 91403 Orsay cedex, France

mixSd~limsi, ft.

Abstract.

From the early steps of sketching to final engineering, a fre-

quent and very important activity

designing objects is to perform

graphical and spatial simulations to solve the constraints on the objects

which are being designed. But when we analyse work situations involving

the use of CAD systems, it is today an acknowledged fact that these tools

are not helpful to perform these types of simulations. While knowledge

modeling based on form feature concepts already offers some possibili-

ties for attaching behaviour to objects, the simulation activity requires

addition a 'real time' and 'intelligent' management of the interactions

between the 3D virtual objects and the CAD user.

Our general purpose is to study how future CAD systems could be

improved to achieve the simulation steps of object design. In this context

we present some issues concerning the cooperation between a model of

reactive 3D objects and a multimodal X Window kernel. We have de-

veloped a prototype of a system where objects with reactive behaviour

can

be built, and with which the user can interact with a combination of

graphical actions and vocal commands. This prototype is used to evaluate

the feasability and the usefulness of the integration of such techniques

in futur applications that would be used by object designers in a real

working context. We describe the current state of this system and the

planned improvements.

1 Introduction

One of the most promising results of the research on multimodal user interfaces

is their capacity to transform industrial applications which manipulate 2D or

3D virtual spaces. For example, it has been shown that a user can be more

'productive' using a CAD program when keyboard interactions are replaced by

vocal ones (Martin, 1989). Hence some researchers propose to rethink these tools

in terms of multimodal interaction (Gaildrat et al., 1993).

More than with simple substitutions of modalities added to the traditional

user interface of CAD systems, our interest is to study how multimodal inter-

action could make these systems contribute better to object design. In fact,

object design is not only an engineering activity, for which CAD systems are

already useful. A frequent and very important activity is to perform graphical

and spatial simulations to solve the constraints on the objects which are being

Cooperation between Reactive 3D Objects and ... 189

designed. Unfortunately, these simulations are not easily performed with current

CAD systems, and the use of such tools for this activity requires complicated

work organisations (Lebahar, 1992). We will hereafter refer to such simulations

as 'object simulations' or '3D simulations'.

In this context, our general purpose is to study how future CAD systems

could be really helpful for object simulation. Indeed, this activity requires a

'real time' and 'intelligent' management of the interactions between the 3D vir-

tual objects and/or between these objects and the CAD user. An object (already

built by the user or under construction) must have short-delay reactive behaviour

in relation to the other objects of the virtual space it has dependencies with.

Additionally, the user's interactions with these virtual objects must be efficient,

requiring advanced combinations of modalities both for input and output, syn-

tactic and semantic dialogue analysis, and so on.

But reactive behaviour as well as multimodal interaction with 3D virtual

objects presuppose knowledge modeling for these objects. During the last ten

years, knowledge modeling for CAD/CAM is one of the main purposes of the

form feature

approach. According to Shah (1990), "a

form feature

is a physical

constituent of a part with a generic shape realisable or abstract, it has signifi-

cance in design, analysis, manufacturing, or some other engineering domain and

has predictable behaviour or properties". From this perspective, parametrical

and variational modeling concepts of 'form feature' are very close to the require-

ments of a CAD system when used for 2D or 3D object simulations. On the other

hand, we have to take into account that the simulation activity for object design

can start from the first steps of sketching in the design process. In other words,

during some simulation steps the objects might not have sufficient properties to

be attached to form feature classes but only to geometrical semantics. Consid-

ering that 'good' knowledge modeling generally requires a hierarchical strategy,

we chose to focus our work on the simulation activity with geometrical and topo-

logical objects. Indeed, our objective is to construct a 3D modeling kernel for

object simulation that future CAD systems dedicated to particular object design

domains could share.

However, manipulating virtual spaces requires a powerful graphical environ-

ment. Today, many of these applications are developed on UNIX workstations

using X Window as a standard windowing environment, sometimes with addi-

tional graphical hardware. In spite of the various functions it can realise, the X

server only manages, in terms of input modalities, mouse and keyboard events.

In order to support the kind of interactions we require, we have created an

architecture which seamlessly integrates new advanced modalities.

In the remainder of this chapter, we first define some basic concepts of multi-

modality and we show the importance of semantic representations for multimodal

user interfaces. We subsequently present the geometrical and topological model

of MIX 3D

(Multimodal Interactions under X environment with a 319 virtual

space)

and we explain how knowledge modeling in MIX 3D is used to manage

objects simulation. We then present examples of cooperation between reactive

3D objects and multimodal interactions. Finally, we discuss the design of our

190 Patrick Bourdot, Mike Krus, Rachid Gherbi

multimodal interface architecture in the X Window environment and consider

some general requirements for future enhancements.

2 Multimodal Interactions: Main Concepts

2.1 Definitions of Multimodality

Multimodal user interfaces are frequently opposed to multimedia applications

(Bellik et al., 1995a; Schomaker et al., 1995). The task of a multimedia appli-

cation is to allow users to manage large volumes of information represented by

means of several different media (images, texts, sounds, etc.). The task of a

multimodal user interface is at the input side to analyse, recognise and merge

information coming from several devices, and on the output side to determine

an appropriate combination of media to deliver a message.

Using distinctions of this kind, a three dimensional classification of multi-

media and multimodal systems was introduced by Nigay and Coutaz (1993).

One dimension describes the

levels of abstraction

of a system, while the two

others represent respectively the

fusion levels

and the

use of modalities

that a

system is able to support. For instance, a multimedia application uses low levels

of abstraction because it does not take into account any semantics of the data

to determine meaning. At the opposite end, a multimodal user interface has

several levels of abstraction from raw data to symbolic representations. These

representations allow the system to perform artificial reasoning and to improve

the human-machine interaction. The 'fusion levels' dimension introduces a dis-

tinction between

independent

and

combined

user interfaces. The former are able

to use one modality to receive or produce an expression, while the latter allow

to build expressions from (resp. with) several input (resp. output) modalities.

In the same way, the 'use of modalities' dimension has two possible values. A

user interface is

sequential

parallel

if a single modality or several modalities

are managed at the same time.

The possibility of combining several modalities which are chronologically

or synchronously processed is thus an important characteristic of multimodal

systems. In the following we study the fusion of input modalities. An input

modality produces monomodal events. By hypothesis, a monomodal event can

be the result of a recognition system associated with the modality's input device.

Furthermore, we will see that the state of some devices and the state of their

associated recognition systems are also useful for the fusion of modalities.

2.2 Temporal Proximity as a Fusion Criterion

The main concept of multimodal systems is the combination of monomodal

events. It is necessary to define some criteria allowing us to decide how and when

it is possible to achieve this combination. For instance, temporally close events

in one or several modalities may have to be merged into the same multimodal

expression. But the detection of this temporal closeness might not be sufficient