Paper: Learning Whom to Trust with MACE

ACL ID N13-1132
Title Learning Whom to Trust with MACE
Venue Annual Conference of the North American Chapter of the Association for Computational Linguistics
Session Main Conference
Year 2013

Non-expert annotation services like Amazon?s Mechanical Turk (AMT) are cheap and fast ways to evaluate systems and provide categor- ical annotations for training data. Unfortu- nately, some annotators choose bad labels in order to maximize their pay. Manual iden- tification is tedious, so we experiment with an item-response model. It learns in an un- supervised fashion to a) identify which an- notators are trustworthy and b) predict the correct underlying labels. We match perfor- mance of more complex state-of-the-art sys- tems and perform well even under adversarial conditions. We show considerable improve- ments over standard baselines, both for pre- dicted label accuracy and trustworthiness es- timates. The latter can be further improved by introducing a prior on model parameters and us...