Paper: A Formal Scheme for Multimodal Grammars

ACL ID C10-2008
Title A Formal Scheme for Multimodal Grammars
Venue International Conference on Computational Linguistics
Session Poster Session
Year 2010
Authors

We present in this paper a formal approach for the representation of multimodal in- formation. This approach, thanks to the to use of typed feature structures and hy- pergraphs, generalizes existing ones (typ- ically annotation graphs) in several ways. It first proposes an homogenous represen- tation of different types of information (nodes and relations) coming from differ- ent domains (speech, gestures). Second, it makes it possible to specify constraints representing the interaction between the different modalities, in the perspective of developing multimodal grammars.