Paper: How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation.

ACL ID D09-1031
Title How well does active learning actually work? Time-based evaluation of cost-reduction strategies for language documentation.
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2009
Authors

Machine involvement has the potential to speed up language documentation. We as- sess this potential with timed annotation experiments that consider annotator exper- tise, example selection methods, and sug- gestions from a machine classifier. We find that better example selection and la- bel suggestions improve efficiency, but ef- fectiveness depends strongly on annota- tor expertise. Our expert performed best with uncertainty selection, but gained lit- tle from suggestions. Our non-expert per- formed best with random selection and suggestions. The results underscore the importance both of measuring annotation cost reductions with respect to time and of the need for cost-sensitive learning meth- ods that adapt to annotators.