Paper: On the Predictability of Human Assessment: when Matrix Completion Meets NLP Evaluation

ACL ID P13-2025
Title On the Predictability of Human Assessment: when Matrix Completion Meets NLP Evaluation
Venue Annual Meeting of the Association of Computational Linguistics
Session Short Paper
Year 2013
Authors

This paper tackles the problem of collect- ing reliable human assessments. We show that knowing multiple scores for each ex- ample instead of a single score results in a more reliable estimation of a system quality. To reduce the cost of collect- ing these multiple ratings, we propose to use matrix completion techniques to pre- dict some scores knowing only scores of other judges and some common ratings. Even if prediction performance is pretty low, decisions made using the predicted score proved to be more reliable than de- cision based on a single rating of each ex- ample.