Paper: Formalism-Independent Parser Evaluation with CCG and DepBank

ACL ID P07-1032
Title Formalism-Independent Parser Evaluation with CCG and DepBank
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 2007

A key question facing the parsing commu- nity is how to compare parsers which use different grammar formalisms and produce different output. Evaluating a parser on the same resource used to create it can lead to non-comparable accuracy scores and an over-optimistic view of parser performance. In this paper we evaluate a CCG parser on DepBank, and demonstrate the difficulties in converting the parser output into Dep- Bank grammatical relations. In addition we present a method for measuring the effec- tiveness of the conversion, which provides an upper bound on parsing accuracy. The CCG parser obtains an F-score of 81.9% on labelled dependencies, against an upper bound of 84.8%. We compare the CCG parser against the RASP parser, outperform- ing RASP by over 5% overall and on the ma- jority o...