Paper: ILR-Based MT Comprehension Test with Multi-Level Questions

ACL ID N07-2020
Title ILR-Based MT Comprehension Test with Multi-Level Questions
Venue Human Language Technologies
Session Short Paper
Year 2007

We present results from a new Interagency Language Roundtable (ILR) based compre- hension test. This new test design presents questions at multiple ILR difficulty levels within each document. We incorporated Arabic machine translation (MT) output from three independent research sites, arbi- trarily merging these materials into one MT condition. We contrast the MT condition, for both text and audio data types, with high quality human reference Gold Standard (GS) translations. Overall, subjects achieved 95% comprehension for GS and 74% for MT, across 4 genres and 3 diffi- culty levels. Surprisingly, comprehension rates do not correlate highly with translation error rates, suggesting that we are measur- ing an additional dimension of MT quality. We observed that it takes 15% more time overall...