Information Theory / Science and Technology
Parent article: Sci-Tech: The Difference Between Data and Knowledge
Shannon entropy
Measure of the average uncertainty or unpredictability of an information source. H = −Σ p(x) log p(x). Quantifies the minimum number of bits needed to encode the average output of the source without loss. Origin: imported from Boltzmann’s statistical thermodynamic entropy (1870s) by Shannon (1948). A perfectly predictable signal has H = 0. A uniform distribution maximizes H.
Related: information, bits, encoding, compression, redundancy Cross-references: Complex Systems / Network Science (thermodynamic entropy of Prigogine) Key work: Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3–4), 379–423.
Kolmogorov complexity (algorithmic complexity)
Length of the shortest program in a universal programming language that can produce a given string. Distinct from Shannon entropy at a crucial point: measures the complexity of a specific string, not the average uncertainty of a source. A truly random string has maximum Kolmogorov complexity (it cannot be compressed); it also has maximum Shannon entropy. A highly structured string can have low Kolmogorov complexity and high Shannon entropy if its generating rule is simple but produces much variety.
Related: Shannon entropy, algorithmic information theory, compression, Chaitin Cross-references: none direct Key work: Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1(1), 1–7.
Noise vs. signal
Noise is not an objective property of a transmission: it is the component of a signal that carries no information relevant to the receiver in the context of their specific question. The same datum can be noise for one receiver and signal for another. This distinction is why “more data” does not imply “more knowledge”: additional data may be noise, and noise raises the cost of extracting the signal.
Related: Shannon entropy, filtering, compression, Goodhart’s Law Cross-references: Artificial Intelligence / Machine Learning Key work: Wiener, N. (1948). Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press.
Graceful degradation (information theory context)
Capacity of a communication or network system to maintain the transmission of the most critical information when the channel or network degrades, sacrificing less essential layers in a predictable way. Example: progressive JPEG encoding, which allows reconstructing a low-quality version of the whole before all data is received. Principle the internet implements in its routing protocol.
Related: resilience, progressive encoding, prioritization, redundancy Cross-references: Systems Engineering, Robotics / Swarm Systems Key work: Leiner, B. M., et al. (2009). A brief history of the Internet. ACM SIGCOMM Computer Communication Review, 39(5), 22–31.
Data vs. knowledge
In Shannon’s technical sense: data is the raw signal; knowledge is the reduction of uncertainty that signal produces for a specific receiver with specific prior knowledge. Parallel to Hayek’s distinction between information dispersed among individual agents and data aggregated in a central office: the latter is not equivalent to the former. Aggregating dispersed knowledge into data destroys the dimension that made it informative.
Related: Shannon entropy, tacit knowledge, signal, epistemic reduction Cross-references: Praxeology / Austrian Political Economy Key work: Hayek, F. A. (1945). The use of knowledge in society. American Economic Review, 35(4), 519–530.
Informational resilience
Capacity of a system to maintain the transmission of critical information and recover from perturbations without requiring a central authority to activate the recovery. The internet was designed with this principle explicitly (ARPANET). Markets exhibit this property emergently: price signals adjust to disruptions and redirect resources continuously, without any authority activating the process.
Related: distributed routing, redundancy, graceful degradation, spontaneous order Cross-references: Complex Systems / Network Science, Praxeology / Austrian Political Economy Key work: Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3–4), 379–423.
References
Boltzmann, L. (1877). Über die Beziehung zwischen dem zweiten Hauptsatze der mechanischen Wärmetheorie und der Wahrscheinlichkeitsrechnung. Wiener Berichte, 76, 373–435.
Hayek, F. A. (1945). The use of knowledge in society. American Economic Review, 35(4), 519–530.
Kolmogorov, A. N. (1965). Three approaches to the quantitative definition of information. Problems of Information Transmission, 1(1), 1–7.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3–4), 379–423, 623–656.
Wiener, N. (1948). Cybernetics: Or Control and Communication in the Animal and the Machine. MIT Press.