Paper: Edit Machines For Robust Multimodal Language Processing

ACL ID E06-1046
Title Edit Machines For Robust Multimodal Language Processing
Venue Annual Meeting of The European Chapter of The Association of Computational Linguistics
Session Main Conference
Year 2006

Multimodal grammars provide an expres- sive formalism for multimodal integra- tion and understanding. However, hand- crafted multimodal grammars can be brit- tle with respect to unexpected, erroneous, or disfluent inputs. Spoken language (speech-only) understanding systems have addressed this issue of lack of robustness of hand-crafted grammars by exploiting classification techniques to extract fillers of a frame representation. In this paper, we illustrate the limitations of such clas- sification approaches for multimodal in- tegration and understanding and present an approach based on edit machines that combine the expressiveness of multimodal grammars with the robustness of stochas- ticlanguage models ofspeech recognition. We also present an approach where the edit operations are traine...