You are here:
Some Notes on Information Theory and Its Applications
Citation:
CABEZAS, H. Some Notes on Information Theory and Its Applications. Presented at Institute for Chemical and Engineering Science, Jurong Island, SINGAPORE, October 22 - 24, 2007.
Impact/Purpose:
To inform the public
Description:
Information is in the purest sense one of the most fundamental quantities in nature, and the study of information can often yield fundamental insights that cannot otherwise be obtained. For example, starting from a form of information arising from the work of the statistician R.A. Fisher (1890-1962) and now widely known as Fisher information, the physicist B. R. Frieden has shown that virtually all known laws of nature have information as an underlying and unifying factor. Fisher information is fundamentally a measure of the information that is obtainable by an observer from a given set of observations. Further, an information analog can be derived for the laws of thermodynamics and for many processes such as mass and energy transport. Hence, starting from the Second Law of Thermodynamics and a form of entropy from the work of L.E. Boltzmann (1844-1906) and now widely known as Boltzmann entropy, we then relate these two results to a form of information entropy from the work of C.E. Shannon (1916-2001). Finally, using the aforementioned results, we demonstrate the relationship of the Second Law of Thermodynamics and entropy to a specific form of Fisher information which is a measure of dynamic order. Dynamic order is simply order for dynamic systems. For example, laminar fluid flow has higher dynamic order than turbulent flow, and the transition from laminar to turbulent flow constitutes a dynamic regime change. Dynamic regime changes are important because like phase transitions for equilibrium systems, they constitute fundamental reorganizations of the manner in which dynamic systems operate. We also discuss in detail the practical calculation of Fisher information from mathematical models and from real data. We illustrate the calculation from mathematical models with several results from model systems with biological food webs and with biological food webs with embedded industrial systems and a crudely represented economy. The calculation of Fisher information from real data is illustrated with some interesting results for the oceanic food web in the Bering strait using the oceanographic data and Hare and Mantua, and for the political stability of nation states using data from the State Instability Task Force. Finally, the application of information theory to industrial processes is discussed and illustrated with the purpose of stimulating future research.