Paper: Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators

ACL ID N10-1057
Title Predicting Human-Targeted Translation Edit Rate via Untrained Human Annotators
Venue Human Language Technologies
Session Main Conference
Year 2010
Authors

In the field of machine translation, automatic metrics have proven quite valuable in system development for tracking progress and meas- uring the impact of incremental changes. However, human judgment still plays a large role in the context of evaluating MT systems. For example, the GALE project uses human- targeted translation edit rate (HTER), wherein the MT output is scored against a post-edited version of itself (as opposed to being scored against an existing human reference). This poses a problem for MT researchers, since HTER is not an easy metric to calculate, and would require hiring and training human an- notators to perform the editing task. In this work, we explore soliciting those edits from untrained human annotators, via the online service Amazon Mechanical Tur...