Paper: An Information-Theoretic Empirical Analysis Of Dependency-Based Feature Types For Word Prediction Models

ACL ID W99-0618
Title An Information-Theoretic Empirical Analysis Of Dependency-Based Feature Types For Word Prediction Models
Venue 2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora
Session Main Conference
Year 1999
Authors
  • Dekai Wu (University of Science and Technology, Clear Water Bay Hong Kong)
  • Zhao Jun (University of Science and Technology, Clear Water Bay Hong Kong; Peking University, Beijing China)
  • Zhifang Sui

Over the years, many proposals have been made to incorporate assorted types of feature in language models. However, discrepancies between training sets, evaluation criteria, algorithms, and hardware environments make it difficult to compare the models objectively. In this paper, we take an information theoretic approach to select feature types in a systematic manner. We describe a quantitative analysis of the information gain and the information redundancy for various combinations of feature types inspired by both dependency structure and bigram structure, using a Chinese treebank and taking word prediction as the object. The experiments yield several conclusions on the predictive value of several feature types and feature types combinations for word prediction, which are expected to provi...