Paper: Of Words, Eyes and Brains: Correlating Image-Based Distributional Semantic Models with Neural Representations of Concepts

ACL ID D13-1202
Title Of Words, Eyes and Brains: Correlating Image-Based Distributional Semantic Models with Neural Representations of Concepts
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2013
Authors

Traditional distributional semantic models ex- tract word meaning representations from co- occurrence patterns of words in text cor- pora. Recently, the distributional approach has been extended to models that record the co- occurrence of words with visual features in image collections. These image-based models should be complementary to text-based ones, providing a more cognitively plausible view of meaning grounded in visual perception. In this study, we test whether image-based mod- els capture the semantic patterns that emerge from fMRI recordings of the neural signal. Our results indicate that, indeed, there is a significant correlation between image-based and brain-based semantic similarities, and that image-based models complement text-based ones, so that the best correlations are a...