Paper: Fusing Eye Gaze with Speech Recognition Hypotheses to Resolve Exophoric References in Situated Dialogue

ACL ID D10-1046
Title Fusing Eye Gaze with Speech Recognition Hypotheses to Resolve Exophoric References in Situated Dialogue
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2010
Authors

In situated dialogue humans often utter lin- guistic expressions that refer to extralinguistic entities in the environment. Correctly resolv- ing these references is critical yet challeng- ing for artificial agents partly due to their lim- ited speech recognition and language under- standing capabilities. Motivated by psycholin- guistic studies demonstrating a tight link be- tween language production and human eye gaze, we have developed approaches that in- tegrate naturally occurring human eye gaze with speech recognition hypotheses to resolve exophoric references in situated dialogue in a virtual world. In addition to incorporat- ing eye gaze with the best recognized spo- ken hypothesis, we developed an algorithm to also handle multiple hypotheses modeled as word confusion networks. Our ...