Paper: Test Collection Selection and Gold Standard Generation for a Multiply-Annotated Opinion Corpus

ACL ID P07-2023
Title Test Collection Selection and Gold Standard Generation for a Multiply-Annotated Opinion Corpus
Venue Annual Meeting of the Association of Computational Linguistics
Session System Demonstration
Year 2007
Authors

Opinion analysis is an important research topic in recent years. However, there are no common methods to create evaluation corpora. This paper introduces a method for developing opinion corpora involving multiple annotators. The characteristics of the created corpus are discussed, and the methodologies to select more consistent testing collections and their corresponding gold standards are proposed. Under the gold standards, an opinion extraction sys- tem is evaluated. The experiment results show some interesting phenomena.