Paper: Inter-annotator Agreement for Dependency Annotation of Learner Language

ACL ID W13-1723
Title Inter-annotator Agreement for Dependency Annotation of Learner Language
Venue Innovative Use of NLP for Building Educational Applications
Session
Year 2013
Authors

This paper reports on a study of inter- annotator agreement (IAA) for a dependency annotation scheme designed for learner En- glish. Reliably-annotated learner corpora are a necessary step for the development of POS tagging and parsing of learner language. In our study, three annotators marked several layers of annotation over different levels of learner texts, and they were able to obtain generally high agreement, especially after dis- cussing the disagreements among themselves, without researcher intervention, illustrating the feasibility of the scheme. We pinpoint some of the problems in obtaining full agree- ment, including annotation scheme vagueness for certain learner innovations, interface de- sign issues, and difficult syntactic construc- tions. In the process, we also develop way...