Paper: Improving the Use of Pseudo-Words for Evaluating Selectional Preferences

ACL ID P10-1046
Title Improving the Use of Pseudo-Words for Evaluating Selectional Preferences
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2010
Authors

This paper improves the use of pseudo- words as an evaluation framework for selectional preferences. While pseudo- words originally evaluated word sense disambiguation, they are now commonly used to evaluate selectional preferences. A selectional preference model ranks a set of possible arguments for a verb by their se- mantic fit to the verb. Pseudo-words serve as a proxy evaluation for these decisions. The evaluation takes an argument of a verb like drive (e.g. car), pairs it with an al- ternative word (e.g. car/rock), and asks a model to identify the original. This pa- per studies two main aspects of pseudo- word creation that affect performance re- sults. (1) Pseudo-word evaluations often evaluate only a subset of the words. We show that selectional preferences should instead be evalua...