📂Thermal Physics

Gibbs' Representation of Entropy

Formulas

Given that the macrostate of a system is in the $i$ state, let’s denote the probability as $P_{i}$. The measured entropy $S$ of this system can be expressed as follows: $$S = - k_{B} \sum_{i} P_{i} \ln P_{i}$$ Here, $k_{B}$ is the Boltzmann constant.

Explanation

Shannon Entropy: When the probability mass function of a discrete random variable $X$ is $p(x)$, the entropy of $X$ is expressed as follows. $$H(X) := - \sum p(x) \log_{2} p(x)$$

Gibbs’ expression of entropy takes a similar form to Shannon entropy as defined in probability information theory and others.

Derivation ¹

• Part 1.

  First law of thermodynamics: $$d U = \delta Q + \delta W$$

  From the definition of entropy, where $\delta Q = T dS$ and $\delta W= - p dV$, we get the following equation, which is another form of the first law of thermodynamics. $$dU = T dS - p dV$$
  Considering the total differential of $U$, we have: $$dU = {{ \partial U } \over { \partial S }} dS + {{ \partial U } \over { \partial S }} dV$$ Comparing only the terms of $dS$ in both equations, we can see that the following holds. $$T = {{ \partial U } \over { \partial S }}$$

  Definition of temperature: $${{1 } \over {k_{B} T}} : = {{ d \ln ( \Omega ) } \over {d E }}$$

  Following the definition of temperature, if $\dfrac{1}{k_{B}} \dfrac{ \partial S }{ \partial U } = \dfrac{ d \ln ( \Omega ) }{ d E }$ and the energy is $U = E$, solving the differential equation yields: $$S = k_{B} \ln \Omega$$ Here, $\Omega$ represents the number of states distinguishable by energy $U$.

• Part 2. $S_{\text{total}} = S + S_{\text{micro}}$

  According to Part 1, if the number of states we can easily observe and distinguish is $X$, then $S$ can be expressed as: $$S = k_{B} \ln X$$ If these $X$ states can be equally composed of $Y$ microstates, then, while it’s impossible to observe the microstates directly, the entropy itself can be expressed as $S_{\text{micro}} = k_{B} \ln Y$. Meanwhile, the entire system actually has $X \times Y = XY$ states, so $S_{\text{total}} = k_{B} \ln XY$. If we write this in a formula, it can be expressed as the sum of observed entropy and the entropy corresponding to the microstates. $$\begin{align*} S_{\text{total}} =& k_{B} \ln XY \\ =& k_{B} \ln X + k_{B} \ln Y \\ =& S + S_{\text{micro}} \end{align*}$$

• Part 3. $P_{i}$

  Let the total number of possible microstates in a system be $N$, and the number of macrostates for the $i$ state be $n_{i}$, then $\sum \limits_{i} n_{i} = N$. Therefore, the probability of the macrostate being in the $i$ state is as follows: $$P_{i} := {{n_{i}} \over {N}}$$

• Part 4.

  However, since we cannot easily calculate the entropy of a microstate, if we take a probabilistic expected value, it would be as follows: $$S_{\text{micro} } = \left< S_{i} \right> = \sum_{i} P_{i} S_{i} = \sum_{i} P_{i} k_{B} \ln n_{i}$$ Meanwhile, the entropy of the entire system can simply be represented as $S_{\text{total} } = k_{B} \ln N$, and since it was $S_{\text{total}} = S + S_{\text{micro}}$ in Part 2, $$S = S_{\text{total} } - S_{\text{micro} } = k_{B} \left( \ln N - \sum_{i} P_{i} \ln n_{i} \right)$$ At this point, because $\ln N = \sum_{i} P_{i} \ln N$, the following holds true: $$\ln N - \sum_{i} P_{i} \ln n_{i} = \sum_{i} P_{i} ( \ln N - \ln n_{i} ) = - \sum_{i} P_{i} \ln P_{i}$$ To summarize, we obtain the following formula: $$S = - k_{B} \sum_{i} P_{i} \ln P_{i}$$

  ■