Paper: MEANT: An inexpensive high-accuracy semi-automatic metric for evaluating translation utility based on semantic roles

ACL ID P11-1023
Title MEANT: An inexpensive high-accuracy semi-automatic metric for evaluating translation utility based on semantic roles
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2011
Authors

We introduce a novel semi-automated metric, MEANT, that assesses translation utility by match- ing semantic role fillers, producing scores that cor- relate with human judgment as well as HTER but at much lower labor cost. As machine transla- tion systems improve in lexical choice and flu- ency, the shortcomings of widespread n-gram based, fluency-oriented MT evaluation metrics such as BLEU, which fail to properly evaluate adequacy, become more apparent. But more accurate, non- automatic adequacy-oriented MT evaluation metrics like HTER are highly labor-intensive, which bottle- necks the evaluation cycle. We first show that when using untrained monolingual readers to annotate se- mantic roles in MT output, the non-automatic ver- sion of the metric HMEANT achieves a 0.43 corre- lationcoeffic...