Paper: Using A Randomised Controlled Clinical Trial To Evaluate An NLG System

ACL ID P01-1057
Title Using A Randomised Controlled Clinical Trial To Evaluate An NLG System
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2001
Authors

The STOP system, which generates personalised smoking-cessation letters, was evaluated by a randomised con- trolled clinical trial. We believe this is the largest and perhaps most rigorous task effectiveness evaluation ever per- formed on an NLG system. The de- tailed results of the clinical trial have been presented elsewhere, in the med- ical literature. In this paper we discuss the clinical trial itself: its structure and cost, what we did and did not learn from it (especially considering that the trial showed that STOP was not effective), and how it compares to other NLG eval- uation techniques.