Paper: Automatically Evaluating Answers To Definition Questions

ACL ID H05-1117
Title Automatically Evaluating Answers To Definition Questions
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2005

Following recent developments in the au- tomatic evaluation of machine translation and document summarization, we present asimilarapproach, implementedinamea- sure called POURPRE, for automatically evaluatinganswerstodefinitionquestions. Until now, the only way to assess the cor- rectness of answers to such questions in- volves manual determination of whether an information nugget appears in a sys- tem’s response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003 and TREC 2004 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.