Paper: Survey Of The Message Understanding Conferences

ACL ID H93-1011
Title Survey Of The Message Understanding Conferences
Venue Human Language Technologies
Session Main Conference
Year 1993

In this paper, the Message Understanding Conferences are reviewed, and the natural language system evaluation that is underway in preparation for the next conference is described. The role of the conferences in the evaluation of information extraction systems is assessed in terms of the purposes of three broad classes of evaluation: progress, adequacy, and diagnostic. The conferences have measured system performance primarily to assess progress and the state of the art, but they have also been influenced by the concerns associated with assessing adequacy and providing diagnostics. Challenges for the future of similar evaluations are also discussed.