Paper: Topical Segmentation: a Study of Human Performance and a New Measure of Quality.

ACL ID N12-1022
Title Topical Segmentation: a Study of Human Performance and a New Measure of Quality.
Venue Annual Conference of the North American Chapter of the Association for Computational Linguistics
Session Main Conference
Year 2012
Authors

In a large-scale study of how people find top- ical shifts in written text, 27 annotators were asked to mark topically continuous segments in 20 chapters of a novel. We analyze the re- sulting corpus for inter-annotator agreement and examine disagreement patterns. The re- sults suggest that, while the overall agree- ment is relatively low, the annotators show high agreement on a subset of topical breaks ? places where most prominent topic shifts occur. We recommend taking into account the prominence of topical shifts when evalu- ating topical segmentation, effectively penal- izing more severely the errors on more impor- tant breaks. We propose to account for this in a simple modification of the windowDiff metric. We discuss the experimental results of evaluat- ing several topical segmenter...