Paper: Evaluating Discourse Processing Algorithms

ACL ID P89-1031
Title Evaluating Discourse Processing Algorithms
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 1989
  • Marilyn A. Walker (Hewlett Packard Laboratories, Bristol UK; University of Pennsylvania, Philadelphia PA)

In order to take steps towards establishing a method- ology for evaluating Natural Language systems, we conducted a case study. We attempt to evaluate two different approaches to anaphoric processing in dis- course by comparing the accuracy and coverage of two published algorithms for finding the co-specifiers of pronouns in naturally occurring texts and dia- logues. We present the quantitative results of hand- simulating these algorithms, but this analysis natu- rally gives rise to both a qualitative evaluation and recommendations for performing such evaluations in general. We illustrate the general difficulties encoun- tered with quantitative evaluation. These are prob- lems with: (a) allowing for underlying assumptions, (b) determining how to handle underspecifications, and (c) evaluati...