Paper: Offspring from Reproduction Problems: What Replication Failure Teaches Us

ACL ID P13-1166
Title Offspring from Reproduction Problems: What Replication Failure Teaches Us
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2013
Authors

Repeating experiments is an important in- strument in the scientific toolbox to vali- date previous work and build upon exist- ing work. We present two concrete use cases involving key techniques in the NLP domain for which we show that reproduc- ing results is still difficult. We show that the deviation that can be found in repro- duction efforts leads to questions about how our results should be interpreted. Moreover, investigating these deviations provides new insights and a deeper under- standing of the examined techniques. We identify five aspects that can influence the outcomes of experiments that are typically not addressed in research papers. Our use cases show that these aspects may change the answer to research questions leading us to conclude that more care should be taken in in...