Paper: Quantitative And Qualitative Evaluation Of Darpa Communicator Spoken Dialogue Systems

ACL ID P01-1066
Title Quantitative And Qualitative Evaluation Of Darpa Communicator Spoken Dialogue Systems
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2001
Authors

This paper describes the application of the PARADISE evaluation framework to the corpus of 662 human-computer dialogues collected in the June 2000 Darpa Communicator data collection. We describe results based on the stan- dard logfile metrics as well as results based on additional qualitative metrics derived using the DATE dialogue act tagging scheme. We show that per- formance models derived via using the standard metrics can account for 37% of the variance in user satisfaction, and that the addition of DATE metrics im- proved the models by an absolute 5%.