Paper: Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk

ACL ID W10-1006
Title Rethinking Grammatical Error Annotation and Evaluation with the Amazon Mechanical Turk
Venue Innovative Use of NLP for Building Educational Applications
Session
Year 2010
Authors

In this paper we present results from two pi- lot studies which show that using the Amazon Mechanical Turk for preposition error anno- tation is as effective as using trained raters, but at a fraction of the time and cost. Based on these results, we propose a new evaluation method which makes it feasible to compare two error detection systems tested on different learner data sets.