Paper: Lessons Learned From Large Scale Evaluation Of Systems That Produce Text: Nightmares And Pleasant Surprises

ACL ID W06-1401
Title Lessons Learned From Large Scale Evaluation Of Systems That Produce Text: Nightmares And Pleasant Surprises
Venue International Conference on Natural Language Generation
Session Main Conference
Year 2006
Authors

As the language generation community explores the possibility of an evaluation program for lan- guage generation, it behooves us to examine our experience in evaluation of other systems that pro- duce text as output. Large scale evaluation of sum- marization systems and of question answering sys- tems has been carried out for several years now. Summarization and question answering systems produce text output given text as input, while lan- guage generation produces text from a semantic representation. Given that the output has the same properties, we can learn from the mistakes and the understandings gained in earlier evaluations. In this invited talk, I will discuss what we have learned in the large scale summarization evalua- tions carried out in the Document Understanding Conferences (D...