Claude Shannon’s Mathematical Conceptualization of Surprise: Information Theory
Claude Shannon, often called the “father of modern information theory,” would likely approach the concept of surprise through the lens of information content. According to Shannon, the amount of information (or surprise) in receiving a particular message is proportional to the logarithm of the reciprocal of the probability of that message occurring. Mathematically, this can be expressed as:
Defining price through Probability
Shannon’s formula highlights that events with lower probabilities generate higher information content, or greater surprise. If an event is certain (( P = 1 )), then the information content is zero, meaning there is no surprise. Conversely, highly improbable events (( P ) close to 0) result in high information content, signifying a great degree of surprise.
Entropy as Average Surprise
Shannon also introduced the concept of entropy, defined as the average information (or surprise) content expected for a given source of messages. The entropy ( H ) of a discrete random variable ( X ) with possible outcomes ( x_1, x_2, …, x_n ) and corresponding probabilities ( p(x_1), p(x_2), …, p(x_n) ) is given by:
Entropy gives us a measure of the “average surprise” one should expect from a particular source, considering all possible outcomes and their respective probabilities.
Redundancy and Predictability
Shannon also discussed the concept of redundancy in a message source. High redundancy means that the message is very predictable, which, in turn, means that there will be little surprise when the message is received. In contrast, a message with low redundancy is less predictable and thus more surprising.
Conclusion
In summary, Claude Shannon would describe surprise as a function of the information content of an event, which itself is a function of the probability of that event occurring. This mathematical framework offers a precise way to quantify and understand the essence of surprise, linking it closely to the principles of information theory.