Paper: DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement

ACL ID C14-2023
Title DKPro Agreement: An Open-Source Java Library for Measuring Inter-Rater Agreement
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2014
Authors

In this paper, we introduce a novel Java implementation of multiple inter-rater agreement mea- sures, which we make available as open-source software. Besides assessing the reliability of coding tasks using S, pi, ?, ?, etc., we particularly support unitizing tasks by measuring ? U as the agreement of the boundaries of the identified annotation units. We provide a unified interface and data model for both tasks as well as multiple diagnostic devices for analyzing the results.