Paper: Learning when to point: A data-driven approach

ACL ID C14-1189
Title Learning when to point: A data-driven approach
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2014

The relationship between how people describe objects and when they choose to point is complex and likely to be influenced by factors related to both perceptual and discourse context. In this paper, we explore these interactions using machine-learning on a dialogue corpus, to identify multimodal referential strategies that can be used in automatic multimodal generation. We show that the decision to use a pointing gesture depends on features of the accompanying description (especially whether it contains spatial information), and on visual properties, especially distance or separation of a referent from its previous referent.