Why does Hartree-Fock overestimate the energy gap?

Question

Everybody knows that Hartree-fock overestimates the energy or band gap, but it is difficult to have a clear explanation of the origin of this large deviation from the experimental values.

I know that this may come from the bad description of the unoccupied states or directly from the exchange interaction which is used to correct the gap in DFT with hybrid functionals. So my question is to know if there is a clear mathematical, physical or chemical explanation of origin of these large gap values obtained with the Hartree-Fock approximation.

Answer 1

I will try to provide a "intuitive" explanation.

The HF approach seek to find the best set of orbitals that minimizes the energy of the ground state. This procedure leads to a set of optimized occupied orbitals that, in the context of solid state theory, form the valence bands, and a set of unoccupied orbitals forming the conduction band. These unoccupied orbitals are also refer to as virtual orbitals and are "only" a by-product of the orthogonalization procedure. They have no real "physical" meaning.

The energy gap between valence and conduction band in the solid state case is related to the energy difference between the highest occupied molecular orbital (HOMO) and lowest unoccupied (or virtual) molecular orbital in the molecular picture. In addition to electron correlation, the orbital relaxation effects, both not taken into account by the HF approach, contribute to the discrepancy in the calculated energy gap. While it is not straightforward to predict in which direction the electron-correlation effects will shift the energy gap, relaxation effects will more likely lead to a stabilization of the excited state, hence decreasing the energy gap.

One can summarize it as follow: the virtual orbitals that form the valence band are not optimized within a HF calculation, hence their energy is higher, leading to a larger band gap in the solid state case.