Paper: Learning part-of-speech taggers with inter-annotator agreement loss

ACL ID E14-1078
Title Learning part-of-speech taggers with inter-annotator agreement loss
Venue Annual Meeting of The European Chapter of The Association of Computational Linguistics
Session Main Conference
Year 2014
Authors

In natural language processing (NLP) an- notation projects, we use inter-annotator agreement measures and annotation guide- lines to ensure consistent annotations. However, annotation guidelines often make linguistically debatable and even somewhat arbitrary decisions, and inter- annotator agreement is often less than perfect. While annotation projects usu- ally specify how to deal with linguisti- cally debatable phenomena, annotator dis- agreements typically still stem from these ?hard? cases. This indicates that some er- rors are more debatable than others. In this paper, we use small samples of doubly- annotated part-of-speech (POS) data for Twitter to estimate annotation reliability and show how those metrics of likely inter- annotator agreement can be implemented in the loss functions...