Paper: Discrete vs. Continuous Rating Scales for Language Evaluation in NLP

ACL ID P11-2040
Title Discrete vs. Continuous Rating Scales for Language Evaluation in NLP
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2011
Authors

Studies assessing rating scales are very com- mon in psychology and related fields, but are rare in NLP. In this paper we as- sess discrete and continuous scales used for measuring quality assessments of computer- generated language. We conducted six sep- arate experiments designed to investigate the validity, reliability, stability, interchangeabil- ity and sensitivity of discrete vs. continuous scales. We show that continuous scales are vi- able for use in language evaluation, and offer distinct advantages over discrete scales.