Paper: Modeling Annotators: A Generative Approach to Learning from Annotator Rationales

ACL ID D08-1004
Title Modeling Annotators: A Generative Approach to Learning from Annotator Rationales
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2008
Authors

A human annotator can provide hints to a machine learner by highlighting contextual “rationales” for each of his or her annotations (Zaidan et al., 2007). How can one exploit this side information to better learn the desired parameters θ? We present a generative model of how a given annotator, knowing the true θ, stochastically chooses rationales. Thus, observing the rationales helps us infer the true θ. We collect substring rationales for a sentiment classification task (Pang and Lee, 2004) and use them to obtain significant accuracy improvements for each annotator. Our new generative approach exploits the rationales more effectively than our previous “masking SVM” approach. It is also more principled, and could be adapted to help learn other kinds of probabilistic classi- fier...