The Zipf Mystery

How do you express yourself? Through sports? Drawings? Socializing? How about something as simple as speech? Speech has been a way for humans to express themselves ever since organized words have been developed. It is a way to get important ideas across. Without it, we would be a world in chaos. There would be no technology, no government, no religion, no structure. But how much do we really know about the words we speak?

When I think about speech, I think about how structured it is. We plan what we are going to say, a fraction of a second before it leaves our mouths. It could also be spontaneous, and creative… or could it?

George Kingsley Zipf wanted to study human speech patterns and noticed something bizarre about the way that we talk. Jonathan Harris is an artist and a computer scientist. He is known for his work with data and storytelling. In 2003, he created a list of 86,800 words from the least to most commonly used in the British National Corpus. The most commonly used words in order are: The, of, and, to, a, in, is, I, that, it, for, you, was, with, on, has, have, but, be, they. The Zipf law notices a pattern between the words. It is noticed in books, articles, or entire languages (even ancient one’s we haven’t fully translated yet) (Newman, MEJ).

The second most commonly used word in the sequence will show up about half as much as the most commonly used, the third one-third as much, the fourth one-fourth as much, the fifth one-fifth and so on and so forth the less common a word you have. In other words, the amount of times that a word is used (frequency) is proportional to one over its rank. (Newman, MEJ)

The algebraic equation comes out to look something like this:

f(r) ∝ 1/r α (Steven T.)

The most popular theory as to why this happens in language is that  language developed speakers preferred to use as few words as possible to get their ideas out. But, in order to understand what was being said, the listeners preferred longer vocabularies that were more specific. The compromise between listening and speaking lead us to our speech patterns that we use today. A few words are used a lot and many words are used rarely. (Newman, MEJ)

Scientists also think that the Zipf Law is a form of the Pareto Principle of Least Resistance. The Pareto Principle explains that because so many things in life behave this way it is worth assuming that 20% of the causes are responsible for 80% of the outcomes. It is called the “ deterministic description of world behavior”.(Newman, MEJ)

Through George Kingsley Zipf”s and Vilfredo Pareto's work we have accelerated our knowledge of linguistics. Everything that we thought we knew about the way we speak has begun to fray. They proved that nothing is the way that it seems. They showed the world that everything can be predicted, no matter how spontaneous it may seem.

Thinking about this only makes me wonder. That something, so complicated, so beautiful, can be portrayed in a way that is so predictable. What else don’t we know about ourselves that we always thought was unique?