Paper: Co-reference via Pointing and Haptics in Multi-Modal Dialogues

ACL ID N12-1058
Title Co-reference via Pointing and Haptics in Multi-Modal Dialogues
Venue Annual Conference of the North American Chapter of the Association for Computational Linguistics
Session Main Conference
Year 2012
Authors

This paper describes our ongoing work on resolving third person pronouns and deictic words in a multi-modal corpus. We show that about two thirds of these referring expressions have antecedents that are introduced by point- ing gestures or by haptic-ostensive actions (actions that involve manipulating an object). After describing our annotation scheme, we discuss the co-reference models we learn from multi-modal features. The usage of haptic- ostensive actions in a co-reference model is a novel contribution of our work.