Paper: Adaptive Language Modeling Using The Maximum Entropy Principle

ACL ID H93-1021
Title Adaptive Language Modeling Using The Maximum Entropy Principle
Venue Human Language Technologies
Session Main Conference
Year 1993
Authors

We describe our ongoing efforts at adaptive statistical language mod- eling. Central to our approach is the Maximum Entropy (ME) Prin- ciple, allowing us to combine evidence from multiple sources, such as long-distance triggers and conventional short.distance trigrams. Given consistent statistical evidence, a unique ME solution is guar- anteed to exist, and an iterative algorithm exists which is guaranteed to converge to it. Among the advantages of this approach are its simplicity, its generality, and its incremental nature. Among its disadvantages are its computational requirements. We describe a succession of ME models, culminating in our current Maximum Likelihood / Maximum Entropy (ML/ME) model. Preliminary re- sults with the latter show a 27% perplexity reduction as compared to a conv...