Paper: Exponential Priors For Maximum Entropy Models

ACL ID N04-1039
Title Exponential Priors For Maximum Entropy Models
Venue Human Language Technologies
Session Main Conference
Year 2004

Maximum entropy models are a common mod- eling technique, but prone to overfitting. We show that using an exponential distribution as a prior leads to bounded absolute discounting by a constant. We show that this prior is better motivated by the data than previous techniques such as a Gaussian prior, and often produces lower error rates. Exponential priors also lead to a simpler learning algorithm and to easier to understand behavior. Furthermore, exponential priors help explain the success of some previ- ous smoothing techniques, and suggest simple variations that work better.