Paper: Analyzing Disagreements

ACL ID W08-1202
Title Analyzing Disagreements
Venue Coling 2008: Proceedings of the 2nd workshop on Information Retrieval for Question Answering
Year 2008

We address the problem of distinguishing between two sources of disagreement in annotations: genuine subjectivity and slip of attention. The latter is especially likely when the classi cation task has a default class, as in tasks where annotators need to nd instances of the phenomenon of inter- est, such as in a metaphor detection task discussed here. We apply and extend a data analysis technique proposed by Beigman Klebanov and Shamir (2006) to rst dis- till reliably deliberate (non-chance) anno- tations and then to estimate the amount of attention slips vs genuine disagreement in the reliably deliberate annotations.