Entropy Measures vs. Kolmogorov Complexity
Abstract
1. Introduction
- Rényi entropy [3], an additive measure based on a specific form of mean of the elementary information gain: instead of using the arithmetic mean, Rényi used the Kolmogorov-Naguno mean associated with the function where and are constants, b is a real greater than 1 and α is a non-negative parameter;
- Tsallis entropy [4], a non additive measure, often called a non-extensive measure, in which the probabilities are scaled by a positive power α, that may either reinforce the large (if ) or the small (if ) probabilities.
2. Preliminaries
2.1. Kolmogorov Complexity
- -
- ;
- -
- and ;
- -
- There are at least c-Kolmogorov random strings of length n.
2.2. Entropies
3. Kolmogorov Complexity and Entropy: How Close?
- , since describing is essentially equivalent to describe , and y;
- ;
- .
- For , ;
- For , .
- For , ;
- For , .
- , where ;
- , where ;
- , where ;
- , where .
- First observe that:Notice also that to describe it is sufficient to give n, so , where c is a real number.By Theorem 2.12, we have,which implies that:On the other hand, by definition:To prove that , it is sufficiently to prove that:i.e., the limit of the following expression is bigger than 0:But,
- To prove this item we use the other inequality of Theorem 2.12:which implies thatSo,Thus, taking n sufficiently large, the conclusion follows.
- The Tsallis entropy of order α of distribution isUsing the inequality 21, we getSince , for n sufficiently large,
- ;
- ;
- ;
- .
4. On the Entropies of the Time-Bounded Universal Distribution
- If , ;
- If , .
5. Uniform Continuity of the Entropies
5.1. Tsallis Entropy
5.2. Shannon Entropy
5.3. Rényi Entropy
6. Conclusions
Acknowledgements
References
- Li, M.; Vitányi, P. An Introduction to Kolmogorov Complexity and Its Applications, 3rd ed.; Springer Publishing Company, Incorporated: New York, NY, USA, 2008. [Google Scholar]
- Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423, 623–656. [Google Scholar] [CrossRef]
- Rényi, A. On measures of entropy and information. In Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability, Statistical Laboratory of the University of California, Berkeley, CA, USA, 20 June–30 July 1960; pp. 547–561.
- Tsallis, C. Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys. 1988, 52, 479–487. [Google Scholar] [CrossRef]
- Cachin, C. Entropy measures and unconditional security in cryptography, 1st ed.; In ETH Series in Information Security and CryptographyHartung-Gorre Verlag: Konstanz, Germany, 1997. [Google Scholar]
- Cover, T.; Gacs, P.; Gray, M. Kolmogorov’s contributions to information theory and algorithmic complexity. Ann. Probab. 1989, 17, 840–865. [Google Scholar] [CrossRef]
- Solomonoff, R. A formal theory of inductive inference; Part I. Inform. Contr. 1964, 7, 1–22. [Google Scholar] [CrossRef]
- Kolmogorov, A.N. Three approaches to the quantitative definition of information. Probl. Inform. Transm. 1965, 1, 1–17. [Google Scholar] [CrossRef]
- Chaitin, G.J. On the length of programs for computing finite binary sequences. J. ACM 1966, 13, 547–569. [Google Scholar] [CrossRef]
- Solovay, R.M. Draft of a paper (or series of papers) on Chaitin’s work. IBM Thomas J. Watson Research Center, Yorktown Heights, New York, NY, USA. Unpublished manuscript. 1975. [Google Scholar]
- Levin, L. Laws of information conservation (non-growth) and aspects of the foundation of probability theory. Prob. Peredachi Inf. 1974, 10, 206–210. [Google Scholar]
- Gacs, P. On the symmetry of algorithmic information. Soviet Mathematics Doklady 1974, 15, 1477–1480. [Google Scholar]
- Grunwald, P.; Vitanyi, P. Shannon information and Kolmogorov complexity. arXiv 2004. arXiv:cs/0410002v1. [Google Scholar]
- Ramshaw, J. Thermodynamic stability conditions for the Tsallis and Rényi entropies. Phys. Lett. A 1995, 198, 119–121. [Google Scholar] [CrossRef]
- Tadaki, K. The Tsallis entropy and the Shannon entropy of a universal probability. In Proceedings of the International Symposium on Information Theory, Toronto, Canada, 6–11 July 2008; abs/0805.0154. pp. 2111–2115.
- Zhang, Z. Uniform estimates on the Tsallis entropies. Lett. Math. Phys. 2007, 80, 171–181. [Google Scholar] [CrossRef]
© 2011 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
Share and Cite
Teixeira, A.; Matos, A.; Souto, A.; Antunes, L. Entropy Measures vs. Kolmogorov Complexity. Entropy 2011, 13, 595-611. https://doi.org/10.3390/e13030595
Teixeira A, Matos A, Souto A, Antunes L. Entropy Measures vs. Kolmogorov Complexity. Entropy. 2011; 13(3):595-611. https://doi.org/10.3390/e13030595
Chicago/Turabian StyleTeixeira, Andreia, Armando Matos, André Souto, and Luís Antunes. 2011. "Entropy Measures vs. Kolmogorov Complexity" Entropy 13, no. 3: 595-611. https://doi.org/10.3390/e13030595