Paper: Visual Information in Semantic Representation

ACL ID N10-1011
Title Visual Information in Semantic Representation
Venue Human Language Technologies
Session Main Conference
Year 2010

The question of how meaning might be ac- quired by young children and represented by adult speakers of a language is one of the most debated topics in cognitive science. Existing semantic representation models are primarily amodal based on information provided by the linguistic input despite ample evidence indi- cating that the cognitive system is also sensi- tive to perceptual information. In this work we exploit the vast resource of images and associ- ated documents available on the web and de- velop a model of multimodal meaning repre- sentation which is based on the linguistic and visual context. Experimental results show that a closer correspondence to human data can be obtained by taking the visual modality into ac- count.