Paper: DATE: A Dialogue Act Tagging Scheme For Evaluation Of Spoken Dialogue Systems

ACL ID H01-1015
Title DATE: A Dialogue Act Tagging Scheme For Evaluation Of Spoken Dialogue Systems
Venue Human Language Technologies
Session Main Conference
Year 2001
Authors

This paper describes a dialogue act tagging scheme developed for the purpose of providing finer-grained quantitative dialogue met- rics for comparing and evaluating DARPA COMMUNICATOR spo- ken dialogue systems. We show that these dialogue act metrics can be used to quantify the amount of effort spent in a dialogue main- taining the channel of communication or, establishing the frame for communication, as opposed to actually carrying out the travel planning task that the system is designed to support. We show that the use of these metrics results in a 7% improvement in the fit in models of user satisfaction. We suggest that dialogue act metrics can ultimately support more focused qualitative analysis of the role of various dialogue strategy parameters, e.g. initiative, across di- alogue sys...