Paper: A chance-corrected measure of inter-annotator agreement for syntax

ACL ID P14-1088
Title A chance-corrected measure of inter-annotator agreement for syntax
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2014
Authors

Following the works of Carletta (1996) and Artstein and Poesio (2008), there is an increasing consensus within the field that in order to properly gauge the reliability of an annotation effort, chance-corrected measures of inter-annotator agreement should be used. With this in mind, it is striking that virtually all evaluations of syntactic annotation efforts use uncor- rected parser evaluation metrics such as bracket F 1 (for phrase structure) and ac- curacy scores (for dependencies). In this work we present a chance-corrected metric based on Krippendorff?s ?, adapted to the structure of syntactic annotations and applicable both to phrase structure and dependency annotation without any modifications. To evaluate our metric we first present a number of synthetic experi- ments to better con...