Paper: Collecting a Why-Question Corpus for Development and Evaluation of an Automatic QA-System

ACL ID P08-1051
Title Collecting a Why-Question Corpus for Development and Evaluation of an Automatic QA-System
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2008
Authors

Question answering research has only recently started to spread from short factoid questions to more complex ones. One significant chal- lenge is the evaluation: manual evaluation is a difficult, time-consuming process and not ap- plicable within efficient development of sys- tems. Automatic evaluation requires a cor- pus of questions and answers, a definition of what is a correct answer, and a way to com- pare the correct answers to automatic answers produced by a system. For this purpose we present a Wikipedia-based corpus of Why- questions and corresponding answers and arti- cles. The corpus was built by a novel method: paid participants were contacted through a Web-interface, a procedure which allowed dy- namic, fast and inexpensive development of data collection methods. Each question...