Paper: A Salience-Based Approach To Gesture-Speech Alignment

ACL ID N04-1004
Title A Salience-Based Approach To Gesture-Speech Alignment
Venue Human Language Technologies
Session Main Conference
Year 2004

One of the first steps towards understanding natural multimodal language is aligning ges- ture and speech, so that the appropriate ges- tures ground referential pronouns in the speech. This paper presents a novel technique for gesture-speech alignment, inspired by salience- based approaches to anaphoric pronoun reso- lution. We use a hybrid between data-driven and knowledge-based mtehods: the basic struc- ture is derived from a set of rules about gesture salience, but the salience weights themselves are learned from a corpus. Our system achieves 95% recall and precision on a corpus of tran- scriptions of unconstrained multimodal mono- logues, significantly outperforming a competi- tive baseline.