Why does the exchange interaction in Hartree-Fock theory lower the total energy?

In Hartree-Fock theory, the expectation value of the total energy can be written as

$$E = \langle\Psi| H |\Psi\rangle = \sum_{a} \langle a| h |a \rangle + \frac{1}{2}\sum_{ab} \big( [aa |bb] - [ab|ba] \big),$$

where the first summation contains the single electron kinetic energy and the interaction energy of the external field. The second summation contains a classical Coulombic term plus an exchange term. The exchange term is introduced by the antisymmetrization of the wavefunction (through a Slater determinant) and takes the form of

$$[ab|ba] = \int \mathrm { d } \mathbf { r } _ { 1 } \mathrm { d } \mathbf { r } _ { 2 } \psi _ { a } ^ { * } \left( \mathbf { r } _ { 1 } \right) \psi _ { b } \left( \mathbf { r } _ { 1 } \right) \frac { 1 } { r _ { 12 } } \psi _ { a } ^ { * } \left( \mathbf { r } _ { 2 } \right) \psi _ { b } \left( \mathbf { r } _ { 2 } \right).$$

This integral is always positive, and so the exchange contribution to the total energy is always negative. Can anyone explain why the exchange contribution to the total energy is negative? In Fermi-Dirac statistics, we can see that the effect of Pauli exclusion of indistinguishable Fermions is an effective outward pressure. Hartree-Fock theory seems to suggest that the exchange interaction lowers the energy of the system, and therefore stabilizes it. Shouldn't the effect of exchange, therefore, be an effective inward pressure?

• Would you check if this is answered by chemistry.stackexchange.com/a/61185/39153 ? It's quite a bit to read, but the relevant bit is in there: the $[aa|bb]$ term is valid for a Hartree product - it intuitively cannot be valid on its own for a Slater determinant. The latter disallows some combinations of orbitals/orbital forms because it does account for Pauli exclusion. $[aa|bb]$ does not accurately calculate the resulting energy by itself, so $[ab|ba]$ must be included to correctly lower the energy. – TAR86 Apr 10 '19 at 5:38
• I understand how the mathematics of a Slater determinant give rise to the form of the exchange integrals. I am having difficulty reconciling the fact that if we used a Hartree product, which is equivalent to removing exchange antisymmetry/Pauli exclusion, then the total energy is higher. This suggests bonding would be weaker if Fermion exchange antisymmetry did not exist. How could the same principle that is responsible for stronger bonds also cause an effective outward pressure in Fermi statistics? – CuriousChemStudent Apr 10 '19 at 19:24
• @CuriousChemStudent I believe the common explanation of why exchange lowers the energy is that it allows the electron motion to be somewhat more correlated. Same spin electrons will avoid each other due to the exchange interaction, reducing the unfavorable electron-electron repulsion and thus lowering the energy. – Tyberius Apr 13 '19 at 1:55
• Thank you, I get it now! There is an effective outward pressure created by a reduction in effective volume cause by Pauli exchange. The volume that is excluded is entirely the highest energy configurations of phase space. Therefore, the electrons correlated by Pauli exchange do not on average exist in locations of high energy with respect to each other. This causes an effective reduction in energy. – CuriousChemStudent Apr 17 '19 at 6:20
• @CuriousChemStudent that's how I've understood it. You could convert your comment to an answer with a bit more detail and I think it would be well received. – Tyberius May 15 '19 at 13:36

Can anyone explain why the exchange contribution to the total energy is negative?

I find it misleading that exchange interaction is treated as something that changes total energy of the system. This lowering of energy is actually due to the Hartree-Fock scheme being in principle inexact, and is not really specific to indistinguishable particles.

Let me elaborate. Consider a hypothetical system of distinguishable particles with exactly the same masses, charges and spins (and all their other properties entering the Schrödinger's equation). Let these parameters be equal to those of an electron. Now, if you solve the Schrödinger's equation for this system in the electromagnetic field of an atomic nucleus, you'll get some (spinor-valued) wavefunctions $$\Psi_n$$, where $$n$$ enumerates the states. These wavefunctions won't in general be antisymmetric.

But the symmetry of the Hamiltonian, namely its invariance with respect to exchange of a pair of particles (since all their relevant properties are the same), tells us that there's a large degeneracy. If you linearly combine the degenerate states corresponding to the same energy so as to get symmetric $$\Psi_k^{\mathrm s}$$ and antisymmetric $$\Psi_k^{\mathrm a}$$ linear combinations, these will still be solutions of the Schrödinger's equation we started with.

If you throw away the symmetric solutions and only leave the antisymmetric ones, these will actually be the solutions for the case where our particles are indistinguishable. You see? We've just arrived at exchange interaction between indistinguishable particles, without any change in energy. What we got is just a possible increase in ground state energy due to losing a bunch of states (and thus renaming one of the formerly excited states to the ground one).

Now, why do we get lowering of states in inexact schemes when we introduce such antisymmetrization? That's directly related (at least for the antisymmetrized ground state) to the variational principle, which tells us that an approximation of a ground state will always have mean energy $$E_{\text{approx}}$$ (which is the Rayleigh quotient) obeying

$$E_{\text{approx}}\ge E_0,$$

where $$E_0$$ is the exact ground state energy.

Before introducing the Slater determinant, the trial wavefunction was smooth and generally nonzero at the loci of electron-electron collisions. But the exact (even non-symmetric) eigenfunction should have a local minimum (a cusp, see here for an example) or a (smooth) zero at such loci due to the Coulomb repulsion. After antisymmetrization, many of such loci of collisions will get zeros. This will make the approximation better, and the mean energy lower, closer to the exact eigenenergy.

If instead of Hartree-Fock method we used some other approximation that had begun with exact treatment of electron collisions, sacrificing something other, then such antisymmetrization might not lead to any improvement in estimation of eigenenergies (limit case being very precise numerical treatment yet disregarding Pauli principle, which is basically equivalent to an exact solution for distinguishable particles).

• The last paragraph reads to me like it contradicts the rest. The fact that exchange leads to zeros for same spin electron collisions (Fermi holes) should change the total energy. – Tyberius Jan 6 at 23:22
• I like this answer, it definitely gave me a different perspective. This same reasoning can then be applied to correlation energy, guaranteeing exact correlation energy is always negative (or zero). So would Bosons correspond to the symmetric results, meaning that a hypothetical system of Bosons would have lower energy than the same system of Fermions? – CuriousChemStudent Jan 6 at 23:56
• @CuriousChemStudent well, if you take the same spinors for bosons as you took for fermions (however unphysical that could be), or if you ignore spin-dependent energy-changing interactions like spin-orbit interaction (or ignore the spin altogether), then yes, the system of bosons will have lower ground state energy. – Ruslan Jan 7 at 7:16
• @Tyberius I've edited the ending. Hopefully it's a bit clearer now. – Ruslan Jan 7 at 17:07

I have been watching this question with keen interest, but wanted to let someone else go for the bounty -- However it expires tomorrow and no one has answered, so I will give my perspective.

Why does the exchange interaction in Hartree-Fock theory lower the total energy?

The answer is given in the question:

This integral is always positive, and so the exchange contribution to the total energy is always negative.

Again, I appreciate that this very simple answer may not be satisfactory for some, which is why I was waiting for someone else to go for the bounty.

A comment by the person who originally asked the question (not the user that offered the bounty) offers an attempt to explain why it might make phenomenological sense for the energy to go down, and I will not repeat that here because it's really them that deserves to get any points that might be awarded for that line of thinking.

However I have a different perspective and it is this: Feel free to try to make phenomenological models or analogies in your head (such there being an outward "pressure" or "reduction in effective volume") if it helps you in any way (maybe to remember things easier, or for your own peace of mind). But in my experience, such "analogies" using "classical physics" thinking (like pressure and forces) do not in quantum mechanics always work the way you would expect. A good example is quantum entanglement, which has absolutely no classical analog; and another example is quantum spin, which is a bit like classical angular momentum but is not precisely the same.

The best answer really, is that the exchange integral is always positive, and in the equation for the energy it has a negative sign, so it will always reduce the total energy.

You can come up with phenomenological ways to conceptualize this, and perhaps someone else will come up with a valid and convincing argument for why that conceptualization does not work, but the bottom line is that the integral is mathematically always positive, so by definition of the "exchange integral", the "exchange interaction" always lowers the energy here..

• Thanks for getting the ball rolling and for your insight! – Karsten Theis Jan 7 at 15:09

I'll try to give my interpretation of the "physical" explanation of why exchange would lower the energy.

For the true wavefunction, the motion of all the electrons should be correlated, with the classical view being that the electrons are avoiding each other to minimize repulsion. With Hartree-Fock, we find an approximate wavefunction by solving for 1-electron functions using the average potential of the other electrons rather than having each electron feel the instantaneous potential of all the others. This eliminates most, but not all of that correlation.

Exchange terms arise due to the Pauli Exclusion Principle and the fact that electrons are fermions. This causes same spin electrons to be correlated, as they can't occupy the same space. While these electrons aren't fully correlated as they would be if you did Full CI, the exchange part of the correlation is reproduced exactly by Hartree-Fock. If not for exchange, these electrons could be located close to each other, which would be highly repulsive and increase the energy. So by avoiding these interactions, the energy is reduced and this correlative effect is what leads exchange to always be negative.

There is a little more in this same vein on Physics SE and the sources therein.

• I would remove the 'somewhat'. The exchange correlation is recovered exactly by HF as you write they cannot occupy the same space. This is why we mix in HF into hybrid DFA and often term it exact exchange. Good and intuitive answer! – Martin - マーチン Jan 6 at 11:47
• @Martin-マーチン I can see how that phrasing would be confusing, I was trying to say that while they don't have the overall correlation from considering the full instantaneous coulomb interactions, they are still partially correlated due to exchange. I'll change it to make it a little clearer. – Tyberius Jan 6 at 21:22