Paper: Deixis And Conjunction In Multimodal Systems

ACL ID C00-1053
Title Deixis And Conjunction In Multimodal Systems
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2000

In order to realize their full potential, multimodal interfaces need to support not just input from multiple modes, but single comnmnds optinmlly distributed across the available input modes. A multimodal language processing architecture is needed to integrate semantic content from the different modes. Johnston 1998a proposes a modular approach to multimodal language processing in which spoken language parsing is completed before lnultimodal parsing. In this paper, I will demonstrate the difficulties this approach faces as the spoken language parsing component is expanded to provide a compositional analysis of deictic expressions. I propose an alternative architecture in which spoken and multimodal parsing are tightly interleaved. This architecture greatly simplifies the spoken language pa...