Paper: An Agreement Measure for Determining Inter-Annotator Reliability of Human Judgements on Affective Text

ACL ID W08-1209
Title An Agreement Measure for Determining Inter-Annotator Reliability of Human Judgements on Affective Text
Venue Coling 2008: Proceedings of the 2nd workshop on Information Retrieval for Question Answering
Session
Year 2008
Authors

An affective text may be judged to be- long to multiple affect categories as it may evoke different affects with varying degree of intensity. For affect classification of text, it is often required to annotate text corpus with affect categories. This task is often performed by a number of hu- man judges. This paper presents a new agreement measure inspired by Kappa co- efficient to compute inter-annotator relia- bility when the annotators have freedom to categorize a text into more than one class. The extended reliability coefficient has been applied to measure the quality of an affective text corpus. An analysis of the factors that influence corpus quality has been provided.