Presenting: Claude Shannon

Engineers and mathematicians spend their careers making machines run faster within known physical limits. Others, realizing we do not even fundamentally understand what the machine is processing, decide to invent a new science from scratch. Claude Elwood Shannon belonged to the second kind.

He was born in 1916 in Petoskey, Michigan, and died in 2001 in Massachusetts, having lived long enough to see the entire world transformed by his ideas [1]. His contributions did not just drive cryptography and computer science forward, they literally defined the architecture of the digital age. He was not a solemn academic; he was an eccentric, playful inventor fascinated by puzzles. Yet his greatest legacy came from radically reorganizing our understanding of a concept that until then was pure philosophy: information.

The work that puts his name in this article was published in 1948, while he was working at Bell Labs. He published a monumental paper titled A Mathematical Theory of Communication [2]. The central idea sounds almost counterintuitive when stated informally: the meaning of a message is completely irrelevant to its mathematical transmission; the only thing that matters is the surprise, or uncertainty, that it reduces. The consequences of that premise built absolutely everything we now call telecommunications and the internet.

That became what we now call information theory, and it changes a surprisingly basic question. Instead of asking how we can make this copper wire carry a clearer voice, we can ask what the universal mathematical limit is for compressing and transmitting any kind of knowledge in the universe.

The engine of curiosity: the life of a juggler

To understand Shannon’s work, you have to understand his playful spirit. Unlike Markov’s argumentative fury or Bayes’ theological mission, Shannon’s engine was pure fun. He was famous in the halls of Bell Labs, and later at MIT, for riding a unicycle up and down the corridors while juggling [3].

Shannon did not study systems to win debates; he studied them because he loved taking things apart. In 1937, at just 21 years old, he wrote his master’s thesis at MIT, showing that Boolean algebra, the mathematics of zeros and ones, true or false, could be mapped perfectly onto electrical relay circuits [4]. That paper is considered today the most important master’s thesis of the twentieth century, since it is the absolute foundation of how every digital computer is designed.

What drove him to study what he studied? The desire to quantify the unquantifiable. During the Second World War, Shannon worked in cryptography, trying to make the communications between Franklin D. Roosevelt and Winston Churchill unbreakable. There he met the British mathematician Alan Turing, and the two shared the intuition that thought, logic, and human messages were not spiritual magic, but measurable mechanical processes. Shannon set out to isolate that fundamental “substance” and measure it [3].

The problem before Shannon

To understand what changed, it helps to recall what telecommunications looked like in the first half of the twentieth century.

Before Shannon, communication was seen as a physical and electrical problem. If you wanted to send a radio signal or a telegram across the ocean and there was “noise” or static, the engineers’ only solution was to shout louder, increasing the signal’s electrical power. “Information” was a vague concept. A poem, a photograph, and the numbers in a bank account were treated as completely different technical problems. There was no unified metric [1], [3].

The problem is that without an exact measure, engineers had no way of knowing whether their systems were efficient or wasting resources. No one knew what the limit of compression was. Shannon wanted something different. He wanted a physics of information, a unit of measurement as undeniable as mass or temperature, that would apply equally to a phone call, to DNA, or to a symphony. And that required separating the “message” from its “meaning.”

The fundamental unit of information

Suppose you have a message to transmit. Shannon mathematically defined the amount of information produced by a source as equal to its entropy:

\[H(X) = -\sum_{i} P(x_i) \log_2 P(x_i)\]

where $X$ is the set of possible messages, $P(x_i)$ is the probability of a specific message, and $H$ is the entropy, the uncertainty.

The important word here is not mathematics. It is uncertainty. Shannon taught us that information is not what you say, but what you could have said. If I tell you “the sun rose in the east,” the probability is 100%, the surprise is zero, and mathematically I have transmitted zero information to you. If you tell me the result of a fair coin flip, you transmit exactly one bit of information to me, the reduction of two possibilities to one.

What this looks like in practice. Shannon popularized the word “bit” (binary digit). He showed that anything, video, text, audio, photos, could be broken down into a string of zeros and ones. We no longer needed one theory for television and another for the telegraph; under Shannon’s theory, everything was simply “data.”

Noise does not mean defeat

This is where the theory truly revolutionized engineering.

Consider a noisy channel, a faulty cable, an unstable Wi-Fi connection. Intuition says that if there is a lot of noise, it is impossible to transmit a message without errors unless you send the signal with infinite energy. But Shannon’s second theorem, the channel capacity theorem, showed the opposite, strikingly [2].

Shannon proved mathematically that as long as the transmission rate stays below the channel’s “capacity,” it is possible to transmit information with zero errors, no matter how much noise there is.

How? Not by shouting louder, but by coding the information intelligently, adding mathematical redundancy, such as error correcting codes.

Physical degradation is not the final fate of data. If you package information with the right mathematics, the receiver can reconstruct the perfect, crystal clear message out of a heavily corrupted signal.

The limits of the idea and the questions it left open

Shannon’s work was so complete that it nearly solved communications engineering in one stroke, but it opened enormous philosophical doors:

The ghost of meaning (semantics). Shannon made it very clear that his theory did not concern itself with meaning. A gigabyte of pure static noise has more “Shannon information,” entropy, surprise, than a gigabyte of the complete works of Shakespeare, which are predictable and redundant. How do we develop a mathematical theory that measures the usefulness or meaning of information? (This is exactly what Kolmogorov and others would try to address years later.)
The physical cost of information. Shannon linked information to thermodynamic entropy. Years later, Rolf Landauer, with Landauer’s principle, showed that erasing one bit of information always requires a minimum amount of energy [5]. What is the ultimate thermodynamic limit of computation in the universe?
Quantum information. Shannon’s bit is binary, 0 or 1. Today physicists face the qubit, which can be both at once. Will Shannon’s theorems survive intact in a world of entangled quantum communication?

When the system hits the limit

If Bayes’ theorem governs how we learn and PageRank governs how we organize the web, the Shannon limit dictates how we build the world’s infrastructure.

When you browse on 5G, when you stream a movie on Netflix, or when the Voyager 1 probe sends images from outside our solar system with a laughably weak transmitter, all of those systems are operating very close to the Shannon limit. Modern engineers no longer try to invent new laws of communication; they simply build mathematical algorithms, like polar codes or LDPC codes, to try to get as close as possible to the mathematical perfection Shannon calculated in 1948.

In a nutshell: the gossip on Claude Shannon

If we had to sum up this whole story informally:

Claude Shannon was a playful genius who literally rode a unicycle around the office juggling. While every serious engineer of the time was sweating over keeping phone lines free of static, Shannon, who could not stand inefficiency, basically said: “Hey, you are all looking at this wrong. It does not matter if it is voice, text, or video, everything in the universe can be turned into ‘Yes or No’ answers (zeros and ones).”

In doing that, he invented the “bit.” He wrote a formula proving that, mathematically, you can send a file through the worst noise in the world and have it arrive perfect if you lock it up with the right mathematical trick. Shannon did not care much for fame or money; he liked building silly contraptions, like the “Ultimate Machine,” a wooden box with a switch that, when flipped on, popped open a mechanical hand that reached out, flipped the switch back off, and tucked itself away again.

By playing around and building strange gadgets, this one man invented the entire mathematical structure that makes zip files, MP3s, satellites, and the whole internet possible.

References

[1] J. Soni and R. Goodman, A Mind at Play: How Claude Shannon Invented the Information Age. New York, NY: Simon & Schuster, 2017.

[2] C. E. Shannon, “A Mathematical Theory of Communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379 to 423, Jul. 1948.

[3] W. Isaacson, The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution. New York, NY: Simon & Schuster, 2014.

[4] C. E. Shannon, “A Symbolic Analysis of Relay and Switching Circuits,” Transactions of the American Institute of Electrical Engineers, vol. 57, no. 12, pp. 713 to 723, Dec. 1938.

[5] R. Landauer, “Irreversibility and Heat Generation in the Computing Process,” IBM Journal of Research and Development, vol. 5, no. 3, pp. 183 to 191, Jul. 1961.