Paper: On Coreference Resolution Performance Metrics

ACL ID H05-1004
Title On Coreference Resolution Performance Metrics
Venue Conference on Empirical Methods in Natural Language Processing
Session Main Conference
Year 2005
  • Xiaoqiang Luo (IBM T.J. Watson Research Center, Yorktown Heights NY)

The paper proposes a Constrained Entity- Alignment F-Measure (CEAF) for evaluating coreference resolution. The metric is com- puted by aligning reference and system entities (or coreference chains) with the constraint that a system (reference) entity is aligned with at most one reference (system) entity. We show that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm. Comparative exper- iments are conducted to show that the widely- known MUC F-measure has serious flaws in evaluating a coreference system. The proposed metric is also compared with the ACE-Value, the official evaluation metric in the Automatic Content Extraction (ACE) task, and we con- clude that the proposed metric possesses some properties such as symmetry and better...