Why are degenerate orbitals (restricted to a single spin) less stable when neither fully filled nor completely empty? Why, in most molecules, are half-filled and fully-filled shells more stable than partially filled shells? I understand that transition metals often promote electrons to their d orbital to make it half filled because this makes them "more stable." Why would this be the case? Relatedly, why are orbitals filled with single electrons in one spin before doubling up to form pairs and complete orbitals? Why must degenerate orbitals follow hund's rule? It seems like it would be more stable for electrons to pair up as they are added to an orbital rather than to become additional radicals in the orbital. Is hund's rule a result of maximizing stability, or is there some other principle at work?
2 Answers
In short, the so-called Fermi correlation which is a consequence of the Pauli principle and applies only to electrons of the like spin keeps such electrons further apart comparing to the case of electrons of unlike spin. Naively one might think that a larger average distance between electrons of like spin reduces electron-electron repulsion energy which is the reason for the stability of high spin states. And while it is true that for each and every pair of electrons electron-electron repulsion energy is decreased as the average distance increases, it was shown that in the whole electron-electron repulsion energy for the molecular system in a high spin state is actually greater that in a lower spin spate.
So, the modern explanation of the lower energy of high spin states is that a larger average distance between electrons also effectively leads to less screening of the nucleus, i.e. each electron experiences a larger nuclear effective charge. As a result the unpaired electrons can approach the nucleus more closely and the electron-nuclear attraction is increased. And this increase in the electron-nuclear attraction energy (which is negative) overcomes the above mentioned increase in the electron-electron repulsion energy (which is positive) leading to a greater stability of high spin states.
For details, see Boyd, R. J. Nature 1984, 310, 480-481, or any modern textbook on quantum chemistry.
I have a quantum exam on monday, and this is one of the concepts being tested. This is an attempt to justify my procrastination, and an attempt at convincing myself that this counts as review. I am writing this to complement @Wildcat's already excellent answer.
The point that I am trying to prove here is that in general, for a given electron configuration a state in which the spins are unpaired has lower total energy than one in which the spins are paired.
To make this point, I consider the case of an excited helium atom with the electron configuration $1\mathrm{s}^1 2\mathrm{s}^1$
The possible spin pairings:
$$\begin{array}{cc} \hline 1\mathrm{s} & 2\mathrm{s} \\ \hline \uparrow & \uparrow\\ \uparrow & \downarrow \\ \downarrow & \uparrow \\ \downarrow & \downarrow\\ \hline \end{array}$$
This gives rise to $^1S$ and $^3S$ terms (ignoring spin-orbit interaction).
For the singlet case and triple case (assuming the orbital approximation, wherein the exact many electron wavefunction is approximated as a product of one electron wavefunctions. In that, sense $1s$ , $2s$ etc. given below are similar to hydrogenic atomic orbitals but aren't exactly the same. On second thought, I think the notation can be misleading.),
$$\psi_{\text{singlet}} \approx \frac{1}{\sqrt2}[1s(1)2s(2) + 2s(1)1s(2)]\frac{1}{\sqrt2}[\alpha(1)\beta(2)-\beta(1)\alpha(1)]$$
$$\psi_{\text{triplet}} \approx \frac{1}{\sqrt2}[1s(1)2s(2)- 2s(1)1s(2)]\times \left\{ \begin{array}{ll} \alpha(1)\alpha(2) \text{or} \\ \beta(1)\beta(2) \text{or} \\ \frac{1}{\sqrt2}[\alpha(1)\beta(2) + \beta(1)\alpha(2)] \\ \end{array} \right.$$
You can check that these wavefunctions have been construted to obey Pauli's Exclusion Principle.
Notation: $1s(.), 2s(.)$ etc. are the spatial orbitals and $\alpha(.), \beta(.)$ are the spin functions. The number in parantheses is a label for the electron. For example, $1s(1)$ means electron one is in the $1s$ orbital, and $\alpha(1)$ means electron one is spin up.
The Hamiltonian is $$ \hat{H} = \hat{H}_1+ \hat{H}_2 + \frac{e^2}{4 \pi \epsilon_0 r_{12}}$$
where, $\hat{H}_i$ is the hamiltonian for the single elctron $i$ neglection repulions. The third term accounts for inter electron repulsions in this 2 electron case.
$$E = \langle \psi|\hat{H}|\psi \rangle $$
and using the approximate expressions we calculated above to find the energy for the singlet and triplet case. I have dropped the spin functions because the hamiltonian doesn't depend on spin here.
$E_{\text{singlet}} = \frac{1}{2}\int \int ([1s(1)2s(2) + 2s(1)1s(2)])^* \left ( \hat{H}_1+ \hat{H}_2 + \frac{e^2}{4 \pi \epsilon_0 r_{12}} \right)([1s(1)2s(2) + 2s(1)1s(2)]) \mathrm{d} \tau_1 \mathrm{d} \tau_2$
$E_{\text{singlet}} = E_{1s} + E_{2s} + \frac{1}{2}\int \int \mathrm{d} \tau_1 \mathrm{d} \tau_2 ([1s(1)2s(2) + 2s(1)1s(2)])^* \left( (\frac{e^2}{4 \pi \epsilon_0 r_{12}}) \right )([1s(1)2s(2) + 2s(1)1s(2)]) $
$E_{is}$ is simply shorthand for $\langle \phi_{is}|\hat{H}_i|\phi_{is} \rangle$
Now, the last integral breaks up into two integrals
$$J_{12} = \frac{e^2}{8 \pi \epsilon_0} \int \int \mathrm{d} \tau_1 \mathrm{d} \tau_2 [1s(1)]^2 \left(\frac{1}{r_{12}} [2s(2)]^2\right)$$
This is called the Coulomb integral. The second is called the exchange integral and is given below.
$$K_{12} = \frac{e^2}{8 \pi \epsilon_0} \int \int \mathrm{d} \tau_1 \mathrm{d} \tau_2 [1s(1)2s(2)] \left(\frac{1}{r_{12}} [1s(2)2s(1)]\right)$$
Both these integrals are positive quantities. You can convince yourself of this fact if you examine the integrands closely.
Based on this $$E_{\text{singlet}} = E_{1s} + E_{2s}+J_{12}+K_{12}$$ A similar calculation for the triplet state yields (I am omiting the tedious algebra),
$$E_{\text{triplet}} = E_{1s} + E_{2s}+J_{12}-K_{12}$$
the $E_{is}$'s are negative terms (they are just orbital energies). The two positive terms $J$ and $K$ come from introducing coulombic interaction and Pauli's exclusion principle in our system.
You can clearly see that the triplet state with has a lower energy. You can generalise this result to conclude that for a given configuration spins unpaired is always lower in energy than spins paired.
The reason is something @Wildcat has already adresssed. Unpaired elctrons lead to poorer screening, and the increased electron-nucleus interaction is what leads to lowering of energy (despite an increased electron-electron repulsion in this case).
