Home, contact          

Research Activities

This page consists of an overview of my research activities and my approach. You will find also my main results (a short description of the contents of my publications) and an overview of my participations to European projects. Here is the PDF file that I presented at the third Young Researchers' Roundtable on Spoken Dialog Systems (Antwerp, Belgium, Sept. 3rd, 2007).
NB: this page has not been updated since 2007: please look at the publications page.
  Curriculum Vitae          
  Research activities          

Research interests

My primary research is the interpretation of natural language and multimodality (pointing gestures together with referential expressions) in human-machine dialogue situations. In this field my interests are the following (the numbers refer to my publications that are presented below):

Short presentation of my publications

1. Landragin, F., Gaiffe, B., Bellalem, N. & Romary, L., Fusion de contraintes pour la synchronisation des modalités et pour la résolution des références dans un énoncé multimodal, In: Colloque sur les Interfaces Multimodales, 10 ans de multimodalité, Grenoble, France, 2000 (4 pages) Presentation of an algorithm for multimodal references resolution, based on the semantic treatment of logical forms. As opposed to my other publications, this algorithm (the first of my Ph.D. work) is highly rooted in formal semantics.
2. Landragin, F., Bellalem, N. & Romary, L., Compréhension automatique du geste et de la parole spontanés en communication homme-machine : apport de la théorie de la pertinence, In: Oralité et gestualité. Interactions et comportements multimodaux dans la communication (actes du colloque ORAGE'01), L'Harmattan, Paris, France, 2001 (4 pages) Synthesis on the problems linked to the Language and Dialogue team's approach for automatic understanding of multimodal actions. Presentation of the main themes of a methodology based on Relevance Theory.
3. Landragin, F., Bellalem, N. & Romary, L., Visual Salience and Perceptual Grouping in Multimodal Interactivity, In: Workshop on Information Presentation and Natural Multimodal Dialogue, Verona, Italy, 2001 (5 pages) Synthesis on the visual salience criteria and on the perceptual grouping of the objects from a visual scene. Exploitation of these two implicit phenomena for the understanding of linguistic and multimodal referring actions.
4. Landragin, F., Bellalem, N. & Romary, L., Referring to Objects with Spoken and Haptic Modalities, In: IEEE International Conference on Multimodal Interfaces (ICMI'02, Pittsburgh, PA), IEEE CS Press, Los Alamitos, CA, 2002 (6 pages) Demonstration of the possibility to adapt the reference domain model to a particular interaction type, the one of the force feedback gesture (haptic gesture) of the MIAMM project.
5. Landragin, F., The Role of Gesture in Multimodal Referring Actions, In: IEEE International Conference on Multimodal Interfaces (ICMI'02, Pittsburgh, PA), IEEE CS Press, Los Alamitos, CA, 2002 (6 pages) Synthesis on the possible roles of pointing gestures, in terms of reference domains delimitation or referents extraction from implicit domains. Proposition of an algorithm for multimodal reference resolution on this basis..
6. Landragin, F., De Angeli, A., Wolff, F., Lopez, P. & Romary, L., Relevance and Perceptual Constraints in Multimodal Referring Actions, In: van Deemter, K. & Kibble, R. (Eds.), Information Sharing: Reference and Presupposition in Language Generation and Interpretation, CSLI Publications, Stanford, CA, 2002 (19 pages) Presentation of a computational adaptation of Relevance Theory to help multimodal reference resolution. A particular attention is provided to the visual context.
7. Landragin, F., Salmon-Alt, S. & Romary, L., Ancrage référentiel en situation de dialogue, Traitement Automatique des Langues 43(2), Hermès-Lavoisier, Paris, France, 2002 (31 pages) Presentation of the integration of visual perception, language, gesture, and task in an unified framework. This journal paper is the core of my Ph.D. work. The outcome is the specification of an algorithm based on contextual subsets modeling into feature structures, and on the interpretation process modeling into the unification of these feature structures.
8. Landragin, F., La saillance comme point de départ pour l'interprétation et la génération, Journée d'étude de l'Association pour le Traitement Automatique des LAngues (ATALA) sur le thème : structure communicative/structure informationnelle, Paris, France, 2003 (4 pages) Characterization of salience, with the identification of visual salience criteria and linguistic salience criteria. This presentation focus on linguistic salience, with the transposition of visual criteria to linguistics and the proposition of a method for salience computation. The link with communicative structure is established.
9. Landragin, F., Une caractérisation de la pertinence pour les actions de référence, In: Dixième conférence sur le traitement automatique des langues (TALN 2003), Batz-sur-Mer, France, 2003 (10 pages) Proposition of a characterization of the relevance of multimodal referential expressions, with a particular interest on the processing effort. This characterization exploits the notion of reference domain, and is more homogeneous and sharp than the one of publication 6.
10. Landragin, F., Clues for the Identification of Implicit in Multimodal Referring Actions, In: Tenth International Conference on Human-Computer Interaction (HCI International 2003, Heraklion, Crete, Greece), Lawrence Erlbaum Associates, Mahwah, NJ, 2003 (5 pages) Synthesis on the problem of the interpretation in multimodal human-machine dialogue, from the implicit identification point of view. Exploitation of the clues given by the utterance of the user and by the visual scene that constitutes the interaction support.
11. Landragin, F. & Romary, L., Referring to Objects Through Sub-Contexts in Multimodal Human-Computer Interaction, In: Seventh Workshop on the Semantics and Pragmatics of Dialogue (DiaBruck'03), Saarbrücken, Germany, 2003 (8 pages) Systematic investigation of referring phenomena in multimodal dialogue. Proposition of a classification of referring modes linked to the existence of reference domains. Specification of an algorithm for the identification of the possible modes for each type of nominal phrases.
12. Gaiffe, B., Landragin, F. & Quignard, M., Le dialogue naturel comme un service dans un contexte multi-applicatif, In: Journée d'étude de l'Association pour le Traitement Automatique des LAngues (ATALA) sur les relations entre systèmes multi-agents et traitement automatique des langues (AGENTAL), Paris, France, 2004 (10 pages) Investigation of a multi-agent architecture to clearly distinguish the dialogue aspects (one agent: the dialogue manager) from the task-linked aspects (one agent per application). Description of the dialogue manager using three modules: linguistic, collaborative, and applicative. Exploitation of the reference domain model in the linguistic module, with an illustration taken from the OZONE project.
13. Landragin, F., L'utilisation de scores numériques en sémantique computationnelle, Journées scientifiques de Sémantique et Modélisation (JSM'04), Lyon, France, 2004 Synthesis of the various methods to quantify semantic aspects, with the example of salience computation. This presentation completes the publication 8.
14. Landragin, F., Interface sémantique-pragmatique et domaines de référence, In: Quatrièmes Journées d'Études Linguistiques de Nantes (JEL 2004), Nantes, France, 2004 First step towards the integration of verbal semantics to the reference domain model, with the construction of linguistic domains on the basis of predication.
15. Landragin, F., Denis, A., Ricci, A. & Romary, L., Multimodal Meaning Representation for Generic Dialogue Systems Architectures, In: Fourth International Conference on Language Resources and Evaluation (LREC 2004, Lisbon, Portugal), 2004 (4 pages) Description of the update of MMIL (MultiModal Interface Language) for semantic representation, during the re-using of MIAMM architecture to the OZONE demonstrator.
16. Landragin, F. & Romary, L., Dialogue History Modelling for Multimodal Human-Computer Interaction, In: Eighth Workshop on the Semantics and Pragmatics of Dialogue (Catalog'04), Barcelona, Spain, 2004 (8 pages) Synthesis on the design and the management of a dialogue history inside a multimodal human-machine dialogue system. Description of the nature of the modality-linked local histories, and, following publication 15, of the unified representation of information in the histories.
17. Landragin, F., Saillance physique et saillance cognitive, Cognition, Représentation, Langage (CORELA) 2(2), 2004 (24 pages) List of the factors that contribute to visual salience and to linguistic salience, with a distinction between physical factors that come from the physical environment, and cognitive factors that depend on mental states of the subject. The notions of P-salience (physical salience) and C-salience (cognitive salience) are proposed in this way.
18. Landragin, F., Dialogue homme-machine multimodal, Hermès-Lavoisier, Paris, France, 2004, ISBN 2-7462-0992-6 (270 pages) Updated version of my Ph.D. thesis, including three parts. The first part deals with the problems and methods relevant with the resolution of references to objects in human-machine dialogue. The second part presents the concepts and the proposed model based on reference domains. And the third part describes some applications for this model.
19. Landragin, F., Traitement des actes de langage dans un système de dialogue homme-machine, Journées scientifiques de Sémantique et Modélisation (JSM'05), Paris, France, 2005 Synthesis on the phenomena that may be taken into account for speech acts processing in dialogue systems. Focus on complex speech acts, i.e., indirect speech acts and composite speech acts. A comparison is made between complex speech acts processing and collaborativeness in task-oriented dialogue.
20. Landragin, F., Modélisation de la saillance visuelle et linguistique, In: Sixième Colloque des Jeunes Chercheurs en Sciences Cognitives (CJCSC'05), Bordeaux, France, 2005 (6 pages) Proposition of five dimensions or study axes that can be useful when apprehending the notion of salience. The first axis deals with the distinction between visual and linguistic salience, the second one with salience linked to the message form versus the message content, the third one with immediate effect salience versus continuous effect salience, the fourth one with prior salience versus new salience, and the fifth one with informative salience versus rhetorical salience.
21. Landragin, F., Traitement automatique de la saillance, In: Douzième conférence sur le traitement automatique des langues (TALN 2005), Dourdan, France, 2005 (10 pages) Proposition of two principles that characterize the notion of salience and that allow its contextual quantification. The first principle is called primordiality principle and favors the most important entities. The second is called singularity principle and favors the most singular entities, i.e., the entities characterized by the most distinctive features (that the other entities in the context do not have).
22. Landragin, F., Modeling Context for Referring in Multimodal Dialogue Systems, In: Fifth International and Interdisciplinary Conference on Modeling and Using Context (CONTEXT'05), Paris, France, 2005 (14 pages) Synthesis corresponding to the core of my Ph.D. thesis, with the presentation of the multimodal reference domains model. In particular concerning salience and gesture aspects, the contents of publications 3 and 5 are updated and integrated.
23. Landragin, F., Indirect Speech Acts and Collaborativeness in Human-Machine Dialogue Systems, In: First International Symposium on the Exploration and Modelling of Meaning (SEM-05), Biarritz, France, 2005 (8 pages) Following publication 19 and with the aim to make the dialogue systems more collaborative, synthesis on the parameters that a system may take into account when processing indirect and composite speech acts.
24. Landragin, F., Une caractérisation de la référence ostensive indirecte, Revue de Sémantique et de Pragmatique (RSP) 18, Presses Universitaires d'Orléans, France, 2005 (19 pages) List of the phenomena that correspond to deferred ostension, i.e., when the gesture points out an object that is not the referent of the simultaneous linguistic utterance. Classification of the underlying semantic transfers, and presentation of a bottom-up method for the automatic interpretation of these phenomena in multimodal human-machine dialogue.
25. Landragin, F., Visual Perception, Language and Gesture: A Model for their Understanding in Multimodal Dialogue Systems, Signal Processing 86(12), Elsevier, Amsterdam, The Netherlands, 2006 (18 pages) Updated version of the state of the art, approach and concepts from publication 22, together with the algorithm from publication 11. This paper corresponds to the definitive english version of the core of my Ph.D. work.
26. Landragin, F., Modélisation du sens et du contexte sur la base de représentations des objets référés, Journées scientifiques de Sémantique et Modélisation (JSM'06), Bordeaux, France, 2006 This communication questions the feasability of a semantic representation theory that bases on the referring phenomena (which are dominant in dialogue). Some research directions are proposed in order to extend the multimodal reference domains model into such a theory.
27. Landragin, F., Influence de la situation lors de la résolution des anaphores dans le dialogue, Treizième conférence sur le traitement automatique des langues (TALN 2006), Leuven, Belgique, 2006 (10 pages) Starting with the notion of salience and a study of the events that intervene during dialogue, this study consists of a characterization of exophora, i.e., anaphora with no linguistic antecedent but situational antecedent. A method for confronting linguistic and situational parameters is proposed.
28. Sedogbo, C., Grisvard, O., Landragin, F., Lard, J. & Praud, S., HMI Engineering Productivity: the Poor Child of MDE/MDA Trends. A Vision for Model-Driven Human-Computer Interaction Engineering, Dix-huitième Conférence Francophone sur l'Interaction Homme-Machine (Model-Driven Engineering and Human-Computer Interaction Workshop at IHM 2006), Montréal, Canada, 2006 (2 pages) With the statement that it is more and more difficult to design multimodal dialogue systems (due to the increasing complexity of the phenomena we want to address and the increasing complexity of software production), a new method to design systems is proposed. It exploits the principles of model-driven engineering and model-driven architecture (MDE/MDA). The method consists of specifying models for all runtime modules, so that the modules are automatically derived from the models. The main principles of the approach are presented in this abstract and are applied within the EMODE project.
29. Landragin, F., Physical, Semantic and Pragmatic Levels for Multimodal Fusion and Fission, Seventh International Workshop on Computational Semantics (IWCS-7), Tilburg, The Netherlands, 2007 (5 pages) Synthesis on the possible decompositions for the processes for multimodality management, including information fusion and fission. Proposition of three levels (physical, semantic, and pragmatic) with the same kinds of processes in input as well as in output. Statement on the notion of symmetric multimodality.
30. Landragin, F., Taking Situational Factors into Account when Resolving Anaphora: an Approach based on Events and Salience, Sixth Discourse Anaphora and Anaphor Resolution Colloquium (DAARC 2007), Lagos, Portugal, 2007 (6 pages) English version of publication 27, with some additional details dealing with the notion of salience, and some corrections following the comments from the reviewers.
31. Landragin, F., Un exemple de polysémie du geste co-verbal en situation de communication homme-machine et ses conséquences sur les analyses sémantiques et pragmatiques, Deuxième Colloque International de l'Association Française de Linguistique Cognitive (AFLiCo 2), Lille, France, 2007 (2 pages) This presentation is a kind of statement dealing with the famous put-that-there example that is at the origin of a lot of work on multimodal systems. Co-verbal gesture is introduced with various forms, and in particular with the form of a arc of a circle in order to illustrate the moving characteristics of the action. The multimodal fusion process is presented so that the importance of the task is emphasized, for instance when it puts forward some fusion hypotheses and therefore some interpretations.
32. Landragin, F., L'anaphore à antécédent flou : une caractérisation et ses conséquences sur l'annotation des relations anaphoriques, Journée d'étude de l'Association pour le Traitement Automatique des LAngues (ATALA) sur la résolution des anaphores, Paris, France, 2007 (2 pages) Following underspecified semantics, study of some anaphora phenomena, where the precise identification of the antecedent is useless (sometimes impossible). For instance, a pronoun may refer to one antecedent or another with no consequences on the global interpretation of the utterance or text. Introduction of the notion of anaphora with fuzzy antecedent, and characterization of the types of potential fuzzy antecedents. Technical consequences on the annotation of anaphoric relations.

Participations to development projects

NB: More details are available on the French version of this page: Activités de recherche.