Paper: User Studies And The Design Of Natural Language Systems

ACL ID E89-1016
Title User Studies And The Design Of Natural Language Systems
Venue Annual Meeting of The European Chapter of The Association of Computational Linguistics
Session Main Conference
Year 1989

This paper presents a critical discussion of the vari- ous approaches that have been used in the evaluation of Natural Language systems. We conclude that pre- vious approaches have neglected to evaluate systems in the context of their use, e.g. solving a task requir- ing data retrieval. This raises questions about the validity of such approaches. In the second half of the paper, we report a laboratory study using the Wizard of Oz technique to identify NL requirements for carry- ing out this task. We evaluate the demands that task dialogues collected using this technique, place upon a prototype Natural Language system. We identify three important requirements which arose from the task that we gave our subjects: operators specific to the task of database access, complex contextual refer- enc...