Paper: Using Derivation Trees for Informative Treebank Inter-Annotator Agreement Evaluation

ACL ID N13-1061
Title Using Derivation Trees for Informative Treebank Inter-Annotator Agreement Evaluation
Venue Annual Conference of the North American Chapter of the Association for Computational Linguistics
Session Main Conference
Year 2013
Authors

This paper discusses the extension of a sys- tem developed for automatic discovery of tree- bank annotation inconsistencies over an entire corpus to the particular case of evaluation of inter-annotator agreement. This system makes for a more informative IAA evaluation than other systems because it pinpoints the incon- sistencies and groups them by their structural types. We evaluate the system on two corpora - (1) a corpus of English web text, and (2) a corpus of Modern British English.