Paper: Validating the web-based evaluation of NLG systems

ACL ID P09-2076
Title Validating the web-based evaluation of NLG systems
Venue Annual Meeting of the Association of Computational Linguistics
Session Short Paper
Year 2009
Authors

The GIVE Challenge is a recent shared task in which NLG systems are evaluated over the Internet. In this paper, we validate this novel NLG evaluation methodology by comparing the Internet-based results with results we collected in a lab experiment. We find that the results delivered by both methods are consistent, but the Internet- based approach offers the statistical power necessary for more fine-grained evaluations and is cheaper to carry out.