Paper: Differences in User Responses to a Wizard-of-Oz versus Automated System

ACL ID N13-1098
Title Differences in User Responses to a Wizard-of-Oz versus Automated System
Venue Annual Conference of the North American Chapter of the Association for Computational Linguistics
Session Main Conference
Year 2013
Authors

Wizard-of-Oz experimental setup in a dia- logue system is commonly used to gather data for informing an automated version of that system. Previous work has exposed depen- dencies between user behavior towards sys- tems and user belief about whether the sys- tem is automated or human-controlled. This work examines whether user behavior changes when user belief is held constant and the sys- tem?s operator is varied. We perform a post- hoc experiment using generalizable prosodic and lexical features of user responses to a dia- logue system backed with and without a hu- man wizard. Our results suggest that user responses are different when communicating with a wizarded and an automated system, in- dicating that wizard data may be less reliable for informing automated systems than gener- ally a...