Paper: Extension Of Zipf's Law To Words And Phrases

ACL ID C02-1117
Title Extension Of Zipf's Law To Words And Phrases
Venue International Conference on Computational Linguistics
Session Main Conference
Year 2002
Authors

Zipf’s law states that the frequency of word tokens in a large corpus of natural language is inversely proportional to the rank. The law is investigated for two languages English and Mandarin and for n- gram word phrases as well as for single words. The law for single words is shown to be valid only for high frequency words. However, when single word and n-gram phrases are combined together in one list and put in order of frequency the combined list follows Zipf’s law accurately for all words and phrases, down to the lowest frequencies in both languages. The Zipf curves for the two languages are then almost identical.