Paper: Comparing Objective and Subjective Measures of Usability in a Human-Robot Dialogue System

ACL ID P09-1099
Title Comparing Objective and Subjective Measures of Usability in a Human-Robot Dialogue System
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2009
Authors

We present a human-robot dialogue sys- tem that enables a robot to work together with a human user to build wooden con- struction toys. We then describe a study in which na¨ıve subjects interacted with this system under a range of conditions and then completed a user-satisfaction ques- tionnaire. The results of this study pro- vide a wide range of subjective and ob- jective measures of the quality of the in- teractions. To assess which aspects of the interaction had the greatest impact on the users’ opinions of the system, we used a method based on the PARADISE evalua- tion framework (Walker et al., 1997) to de- rive a performance function from our data. The major contributors to user satisfac- tion were the number of repetition requests (which had a negative effect on satisfac- tion),...