Paper: A Machine Learning Based Approach To Evaluating Retrieval Systems

ACL ID N06-1051
Title A Machine Learning Based Approach To Evaluating Retrieval Systems
Venue Human Language Technologies
Session Main Conference
Year 2006
Authors

Test collections are essential to evaluate Information Retrieval (IR) systems. The relevance assessment set has been recog- nized as the key bottleneck in test col- lection building, especially on very large sized document collections. This paper addresses the problem of efficiently se- lecting documents to be included in the assessment set. We will show how ma- chine learning techniques can fit this task. This leads to smaller pools than tradi- tional round robin pooling, thus reduces significantly the manual assessment work- load. Experimental results on TREC col- lections1 consistently demonstrate the ef- fectiveness of our approach according to different evaluation criteria.