Paper: Evaluating Content Selection In Summarization: The Pyramid Method

ACL ID N04-1019
Title Evaluating Content Selection In Summarization: The Pyramid Method
Venue Human Language Technologies
Session Main Conference
Year 2004
Authors

We present an empirically grounded method for evaluating content selection in summariza- tion. It incorporates the idea that no single best model summary for a collection of documents exists. Our method quantifies the relative im- portance of facts to be conveyed. We argue that it is reliable, predictive and diagnostic, thus im- proves considerably over the shortcomings of the human evaluation method currently used in the Document Understanding Conference.