Paper: Besting the Quiz Master: Crowdsourcing Incremental Classification Games

ACL ID D12-1118
Title Besting the Quiz Master: Crowdsourcing Incremental Classification Games
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2012
Authors

Cost-sensitive classification, where the features used in machine learning tasks have a cost, has been explored as a means of balancing knowl- edge against the expense of incrementally ob- taining new features. We introduce a setting where humans engage in classification with incrementally revealed features: the collegiate trivia circuit. By providing the community with a web-based system to practice, we collected tens of thousands of implicit word-by-word ratings of how useful features are for eliciting correct answers. Observing humans? classifi- cation process, we improve the performance of a state-of-the art classifier. We also use the dataset to evaluate a system to compete in the incremental classification task through a reduc- tion of reinforcement learning to classification. Our sy...