Paper: Optimization In Multimodal Interpretation

ACL ID P04-1001
Title Optimization In Multimodal Interpretation
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2004

In a multimodal conversation, the way users communicate with a system depends on the available interaction channels and the situated context (e.g. , conversation focus, visual feedback). These dependencies form a rich set of constraints from various perspectives such as temporal alignments between different modalities, coherence of conversation, and the domain semantics. There is strong evidence that competition and ranking of these constraints is important to achieve an optimal interpretation. Thus, we have developed an optimization approach for multimodal interpretation, particularly for interpreting multimodal references. A preliminary evaluation indicates the effectiveness of this approach, especially for complex user inputs that involve multiple referring expressions in a speech utter...