Paper: Evaluation Challenges In Large-Scale Document Summarization

ACL ID P03-1048
Title Evaluation Challenges In Large-Scale Document Summarization
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2003
Authors

We present a large-scale meta evaluation of eight evaluation measures for both single-document and multi-document summarizers. To this end we built a corpus consisting of (a) 100 Million auto- matic summaries using six summarizers and baselines at ten summary lengths in both English and Chinese, (b) more than 10,000 manual abstracts and extracts, and (c) 200 Million automatic document and summary retrievals using 20 queries. We present both qualitative and quantitative results showing the strengths and draw- backs of all evaluation methods and how they rank the different summarizers.