Paper: Empirical Analysis of Aggregation Methods for Collective Annotation

ACL ID C14-1145
Title Empirical Analysis of Aggregation Methods for Collective Annotation
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2014
Authors

We investigate methods for aggregating the judgements of multiple individuals in a linguistic annotation task into a collective judgement. We define several aggregators that take the relia- bility of annotators into account and thus go beyond the commonly used majority vote, and we empirically analyse their performance on new datasets of crowdsourced data.