Paper: PARADIGM: Paraphrase Diagnostics through Grammar Matching

ACL ID E14-1021
Title PARADIGM: Paraphrase Diagnostics through Grammar Matching
Venue Annual Meeting of The European Chapter of The Association of Computational Linguistics
Session Main Conference
Year 2014

Paraphrase evaluation is typically done ei- ther manually or through indirect, task- based evaluation. We introduce an in- trinsic evaluation PARADIGM which mea- sures the goodness of paraphrase col- lections that are represented using syn- chronous grammars. We formulate two measures that evaluate these paraphrase grammars using gold standard sentential paraphrases drawn from a monolingual parallel corpus. The first measure calcu- lates how often a paraphrase grammar is able to synchronously parse the sentence pairs in the corpus. The second mea- sure enumerates paraphrase rules from the monolingual parallel corpus and calculates the overlap between this reference para- phrase collection and the paraphrase re- source being evaluated. We demonstrate the use of these evaluation metrics on p...