Paper: Textual Properties and Task-based Evaluation: Investigating the Role of Surface Properties Structure and Content

ACL ID W10-4206
Title Textual Properties and Task-based Evaluation: Investigating the Role of Surface Properties Structure and Content
Venue International Conference on Natural Language Generation
Session Main Conference
Year 2010
Authors

This paper investigates the relationship be- tween the results of an extrinsic, task- based evaluation of an NLG system and various metrics measuring both surface and deep semantic textual properties, in- cluding relevance. The latter rely heav- ily on domain knowledge. We show that they correlate systematically with some measures of performance. The core ar- gument of this paper is that more domain knowledge-based metrics shed more light on the relationship between deep semantic properties of a text and task performance.