Paper: EMMA: A novel Evaluation Metric for Morphological Analysis

ACL ID C10-1116
Title EMMA: A novel Evaluation Metric for Morphological Analysis
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2010
Authors

We present a novel Evaluation Metric for Morphological Analysis (EMMA) that is both linguistically appealing and empirically sound. EMMA uses a graph- based assignment algorithm, optimized via integer linear programming, to match morphemes of predicted word analyses to the analyses of a morphologically rich answer key. This is necessary especially for unsupervised morphology analysis systems which do not have access to linguistically motivated morpheme labels. Across 3 languages, EMMA scores of 14 systems have a substantially greater positive correlation with mean average precision in an information retrieval (IR) task than do scores from the metric currently used by the Morpho Challenge (MC) competition series. We compute EMMA and MC metric scores for 93 separate system-language pairs fro...