Entropy in the Physical Sciences

• The Page of Entropy, by Dave Slaven (Physics, Saginaw Valley State University) offers a very nice, non-technical introduction to the statistical notion of physical entropy.
• How to teach statistical physics, by Koo-Chul Lee (Physics, Seoul National University, Korea) offers some Java applets illustrating micrononical ensembles, thermal relaxation, fluctuations in entropy, the Maxwell-Boltmann distribution and other good stuff.

In 1957, E. T. Jaynes discovered a beautiful connection, the Principle of Maximal Entropy, between statistical mechanics and the entropy introduced by Shannon. See:

• A Brief Maxent Tutorial, by Adam Berger (School of Computer Science, Carnegie Mellon University). An html tutorial, featuring some nice geometric insight into the underlying constrained maximization problem.
• Maximal Entropy and Bayesian Probability Theory, a collection of expository sketches and tutorials by someone in the CEMS research group in the Chemical Science and Technology Division, Los Alamos National Laboratory. See especially the excellent tutorial on Inverse Problems and Surprisal Analyisis in Physics.
• Statistical Phasing by Kevin Cowtan (Protein Structure Group, York University). A web tutorial discussing the use of maximal entropy methods in X-ray crystallography.

Information and correlation in statistical mechanical systems, the PhD thesis of David Richard Wolf (Physics, University of Texas, Austin). "In this dissertation the question of how information is carried in a physical system is examined. The systems studied here are simple, as are all systems which have a presentable analysis. The point of view that the states of the physical system of interest may be treated probabilistically, that there is an underlying distribution which describes the probability that a particular state occurs, is taken thoroughly. Certainly the thermodynamic systems studied here are treated on this basis, but more generally whenever such a distribution exists and is known, or is learnable, the methods of this work apply." A particularly interesting aspect of Wolf's work is the introduction of higher order information measures.

In 1975, Stephen Hawking and Jacob Beckenstein discovered an astonishing connection between thermodynamical entropy, quantum mechanics, and general relavitity, which has inspired much current work on quantum gravity. For more information, see:

• The Black Hole Information Loss Problem, by Warren G. Anderson. A very clear explanation of the fundamental problem posed to physics by the existence of Hawking radiation. Part of the fabulous Relavitity FAQ (of which I am a former co-editor).
• The Los Alamos National Laboratory Preprint Server has more than one hundred papers on this subject; you can find them using the search tool (form interface) provided at this site.

The theory of non-equilibrium statistical mechanics began with Lars Onsager (who won a Nobel Prize for related work). For the past several decades, Ilya Prigogine and his coworkers have been developing this theory (Prigogine has also recieved a Nobel Prize for his contributions in this area).

Entropy is closely related to the multifractal "thermodynamical formalism" which has become very popular in the numerical study of dynamical systems, and new entropic quantities are introduced (it seems) daily as physicists (among others) struggle to come to grips with the problem of pinning some kind of meaningful number on dynamical phenomena which are too complex to model in the conventional way. For example, see:

• Local Entropy Characterization of Correlated Random Microstructures, by C. Andraud (Laboratoire d'Optiques des Solides, Universite Pierre et Marie Curie), A. Beghdadi (LPMTM-CNRS, Institut Galilee, Universite Paris Nord), E. Haslund (Institute of Physics, University of Oslo), R. Hilfer (Institute of Physics, University of Oslo and Institut für Physik, Universität Mainz), J. Lafait (Laboratoire d'Optiques des Solides, Universite Pierre et Marie Curie), and B. Virgin (Institute of Physics, University of Oslo).
  

There are some very severe conceptual difficulties in reconciling statistical mechanics with classical thermodynamics. Shortly after Boltzmann claimed to have reduced thermodynamics to the statistics of molecules, Josef Loschmidt and Ernst Zermelo (among others) pointed out still more paradoxes. Some claim that the famous Recurrence Theorem proven in 1898 by Henri Poincare resolved these paradoxes, but I have my doubts, when I have time, I hope to explain my own objections at greater length.

Maxwell himself (who was the first to suggest that thermodynamics might simply be the macroscopic effect of the statistics of molecules) introduced his infamous Demon, who continues to bedevil physicists today. In 1929, Leo Szilard claimed to have resolved this paradox using Heisenberg's Uncertainty Principle. Szilard was one of the first, incidently, to use the word "information" in a scientific paper. In 1982, new insights due to R. Landauer and C. Bennett overturned this paradigm and gave rise to a quite different resolution of Maxwell's paradox. According to Landauer, the erasure of a stored bit inevitably dissipates heat and thus increases entropy. You can read about these ideas in the following paper:

• Maxwell's Demon and the Entropy Cost of Information, by Paul Fahn (Information Systems Laboratory, Stanford University).

Over the years there have been numerous attempts by various people to justify their political or social agenda on the basis of the Second Law of Thermodynamics and other impressive sounding phrases. Here is a paper debunking a recent instance of such entropic pseudoscience:

• The Second Law and Mineral Extraction, by John McCarthy (Computer Science, Stanford University), debunks the idea that low grade ores are useless because of the Second Law.
  

The various aspects of entropy that arise in a variety of applications in statistical physics, thermodynamics and other sciences are given mathematically rigorous and accurate treatment in the book Entropy (edited by Andreas Greven, Gerhard Keller and Gerald Warnecke), which also offers a nice unifying view of the subject.

Further Reading

There are over a hundred statistical mechanics and thermodynamics textbooks, and at least a dozen books have so far been published on the arrow of time alone. Some of the more recent offerings include the following:

• Statistical Dynamics, a stochastic approach to nonequilibrium thermodynamics, by R. F. Streater. Imperial College Press, 1995. Covers entropy in classical statistical dynamics (including the Boltzmann map and Legendre duality) and also in quantum statistical dynamics.
  
• Engines, energy, and entropy: a thermodynamics primer, by John B. Fenn, W.H. Freeman, 1982.
• Thermodynamics and statistical mechanics, by Walter Greiner, Ludwig Neise, and Horst Stocker, Springer-Verlag, 1995. Covers quantum statistical mechanics.
• Time's arrow : the origins of thermodynamic behavior, by Michael C. Mackey, Springer-Verlag, 1992.
• Thermodynamics of chaotic systems,, by C. Beck and F. Schloegl, Cambridge University Press, 1993. A graduate level introduction.
• Complexity : hierarchical structures and scaling in physics, by R. Badii and A. Politi, Cambridge University Press, 1997. Warmly recommended to me by Thierry D. de Wit (Centre de Physique Theorique, Marseille), this book covers entropy, thermodynamical formalism, algorithmic and grammatical complexities, heierarchical scaling complexities, and more. Written at somewhat higher level than the book by Beck and Schloegl.
• Evolution of complex systems: self-organization, entropy, and development, by Rainer Feistel and Werner Ebeling, Kluwer Academic Publishers, 1989.
• The physical basis of the direction of time, by H.-Dieter Zeh, Springer-Verlag, 1989.
• Information and the internal structure of the universe: an exploration into information physics, by Tom Stonier, Springer-Verlag, 1990.
• Entropy, large deviations, and statistical mechanics, by Richard S. Ellis, Springer-Verlag, 1985. Has a good discussion of the connection, via Legendre-Fenchel duality, between the entropy of statistical mechanics and Shannon's entropy.
• Quantum entropy and its use, by Masanori Ohya and Denes Petz. Springer-Verlag, 1993. A research monograph on current efforts to find the proper formulation of entropy in terms of noncommutative C-* algebras.
  

Back to Entropy on the World Wide Web.