Paper: Evaluating Answers To Definition Questions

ACL ID N03-2037
Title Evaluating Answers To Definition Questions
Venue Human Language Technologies
Session Short Paper
Year 2003
  • Ellen M. Voorhees (National Institute of Standards and Technology, Gaithersburg MD)

This paper describes an initial evaluation of systems that answer questions seeking defini- tions. The results suggest that humans agree sufficiently as to what the basic concepts that should be included in the definition of a par- ticular subject are to permit the computation of concept recall. Computing concept precision is more problematic, however. Using the length in characters of a definition is a crude approxi- mation to concept precision that is nonetheless sufficient to correlate with humans’ subjective assessment of definition quality. The TREC question answering track has sponsored a series of evaluations of systems’ abilities to answer closed class questions in many domains (Voorhees, 2001). Closed class questions are fact-based, short an- swer questions. The evaluation of ...