Paper: Pairwise Document Similarity in Large Collections with MapReduce

ACL ID P08-2067
Title Pairwise Document Similarity in Large Collections with MapReduce
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2008
Authors

This paper presents a MapReduce algorithm for computing pairwise document similarity in large document collections. MapReduce is an attractive framework because it allows us to decompose the inner products involved in computing document similarity into separate multiplication and summation stages in a way that is well matched to efficient disk access patterns across several machines. On a col- lection consisting of approximately 900,000 newswire articles, our algorithm exhibits lin- ear growth in running time and space in terms of the number of documents.