Paper: Analysis of Stopping Active Learning based on Stabilizing Predictions

ACL ID W13-3502
Title Analysis of Stopping Active Learning based on Stabilizing Predictions
Venue International Conference on Computational Natural Language Learning
Session Main Conference
Year 2013
Authors

Within the natural language processing (NLP) community, active learning has been widely investigated and applied in or- der to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stopping active learning based on stabilizing predictions (SP). The analysis has revealed three ele- ments that are central to the success of the SP method: (1) bounds on Cohen?s Kappa agreement between successively trained models impose bounds on differences in F-measure performance of the models; (2) since the stop set does not have to be la- beled, it can be made large in practice, helping to guarantee that the results trans- fer to previously unseen streams of ex- amples at test/application time; and (3) good (low va...