Paper: Using The Web To Overcome Data Sparseness

ACL ID W02-1030
Title Using The Web To Overcome Data Sparseness
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2002

This paper shows that the web can be em- ployed to obtain frequencies for bigrams that are unseen in a given corpus. We describe a method for retrieving counts for adjective-noun, noun-noun, and verb- object bigrams from the web by querying a search engine. We evaluate this method by demonstrating that web frequencies and correlate with frequencies obtained from a carefully edited, balanced corpus. We also perform a task-based evaluation, showing that web frequencies can reliably predict human plausibility judgments.