Paper: One Tokenization per Source

ACL ID P98-1076
Title One Tokenization per Source
Venue Annual Meeting of the Association of Computational Linguistics
Session Main Conference
Year 1998
  • Jin Guo (Kent Ridge Digital Labs, Singapore)

We report in this paper the observation of one tokenization per source. That is, the same critical fragment in different sentences from the same source almost always realize one and the same of its many possible tokenizations. This observation is demonstrated very helpful in sentence tokenization practice, and is argued to be with far-reaching implications in natural language processing.