Paper: User-Centered Evaluation Of Interactive Question Answering Systems

ACL ID W06-3007
Title User-Centered Evaluation Of Interactive Question Answering Systems
Venue Interactive Question Answering Workshop
Session
Year 2006
Authors

We describe a large-scale evaluation of four interactive question answering sys- tem with real users. The purpose of the evaluation was to develop evaluation methods and metrics for interactive QA systems. We present our evaluation method as a case study, and discuss the design and administration of the evalua- tion components and the effectiveness of several evaluation techniques with respect to their validity and discriminatory power. Our goal is to provide a roadmap to others for conducting evaluations of their own systems, and to put forward a research agenda for interactive QA evaluation.