Paper: Shared-Task Evaluations In HLT: Lessons For NLG

ACL ID W06-1421
Title Shared-Task Evaluations In HLT: Lessons For NLG
Venue International Conference on Natural Language Generation
Session Main Conference
Year 2006

An alternative is to approach the issue through tasks with inputs and outputs that ‘occur naturally’, so that participants can use their own NLG-specific representations. tations. Examples include data-to-text mappings where e.g. time-series data or a data repository are mapped to fault reports, forecasts, etc. Both data-independent task definitions and tasks with naturally occurring data have promise, but we propose the second as the simpler, easier to organise solution, at least initially. A specific proposal of a set of tasks can be found elsewhere in this volume (Reiter and Belz, 2006). An interesting idea (recommended by ELRA/ELDA) is to break down the input-output mapping into stages (as in the TC-STAR workshops, see table) and then, in a second round of evaluations, to make avai...