Paper: Some Empirical Evidence for Annotation Noise in a Benchmarked Dataset

ACL ID N10-1067
Title Some Empirical Evidence for Annotation Noise in a Benchmarked Dataset
Venue Human Language Technologies
Session Main Conference
Year 2010
Authors

A number of recent articles in computational linguistics venues called for a closer exami- nation of the type of noise present in anno- tated datasets used for benchmarking (Rei- dsma and Carletta, 2008; Beigman Klebanov and Beigman, 2009). In particular, Beigman Klebanov and Beigman articulated a type of noise they call annotation noise and showed that in worst case such noise can severely degrade the generalization ability of a linear classifier (Beigman and Beigman Klebanov, 2009). In this paper, we provide quantita- tive empirical evidence for the existence of this type of noise in a recently benchmarked dataset. The proposed methodology can be used to zero in on unreliable instances, facili- tating generation of cleaner gold standards for benchmarking.