Paper: Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem

ACL ID C14-1208
Title Limitations of MT Quality Estimation Supervised Systems: The Tails Prediction Problem
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2014
Authors

In this paper we address the question of the reliability of the predictions made by MT Quality Estimation (QE) systems. In particular, we show that standard supervised QE systems, usually trained to minimize MAE, make serious mistakes at predicting the quality of the sentences in the tails of the quality range. We describe the problem and propose several experiments to clarify their causes and effects. We use the WMT12 and WMT13 QE Shared Task datasets to prove that our claims hold in general and are not specific to a dataset or a system.