Paper: Intrinsic and Extrinsic Evaluation of an Automatic User Disengagement Detector for an Uncertainty-Adaptive Spoken Dialogue System

ACL ID N12-1010
Title Intrinsic and Extrinsic Evaluation of an Automatic User Disengagement Detector for an Uncertainty-Adaptive Spoken Dialogue System
Venue Annual Conference of the North American Chapter of the Association for Computational Linguistics
Session Main Conference
Year 2012
Authors

We present a model for detecting user dis- engagement during spoken dialogue interac- tions. Intrinsic evaluation of our model (i.e., with respect to a gold standard) yields results on par with prior work. However, since our goal is immediate implementation in a sys- tem that already detects and adapts to user un- certainty, we go further than prior work and present an extrinsic evaluation of our model (i.e., with respect to the real-world task). Cor- relation analyses show crucially that our au- tomatic disengagement labels correlate with system performance in the same way as the gold standard (manual) labels, while regres- sion analyses show that detecting user disen- gagement adds value over and above detecting only user uncertainty when modeling perfor- mance. Our results suggest that ...