Paper: Decoder Integration and Expected BLEU Training for Recurrent Neural Network Language Models

ACL ID P14-2023
Title Decoder Integration and Expected BLEU Training for Recurrent Neural Network Language Models
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2014
Authors

Neural network language models are often trained by optimizing likelihood, but we would prefer to optimize for a task specific metric, such as BLEU in machine trans- lation. We show how a recurrent neural network language model can be optimized towards an expected BLEU loss instead of the usual cross-entropy criterion. Fur- thermore, we tackle the issue of directly integrating a recurrent network into first- pass decoding under an efficient approxi- mation. Our best results improve a phrase- based statistical machine translation sys- tem trained on WMT 2012 French-English data by up to 2.0 BLEU, and the expected BLEU objective improves over a cross- entropy trained model by up to 0.6 BLEU in a single reference setup.