Why large basis sets give better approximations to the exact solution of the Schrödinger equation?

Question

The variational principle states that the energy of any approximate wave function will always be equal to or greater than the energy of the exact solution. Therefore the energy is minimized when performing approximations, in order to get as close as possible to the exact solution. Larger basis sets give the wave equation "more flexibility", but I am not sure I understand this fully. Here's a statistics analogy:

When performing multivariate regression analysis, we choose the regression equation of lowest order; "overfitting" the data is something to be avoided. In my head, choosing large basis sets for simple problems seems like "overfitting".

So why is it that a larger basis set leads to a better approximation? The statistics analogy may not be applicable, but it illustrates what goes on in my head.

Answer 1

The statistics analogy may not be applicable, but it illustrates what goes on in my head.

The overfitting analogy is indeed not applicable here. It is a totally different and unrelated phenomenon. Basically, when we are doing regression analysis we do not know the relationship between a dependent variable and independent variables. Then we can indeed have an overfitting problem when instead of fitting the actual relationship we start to describe some random noise. It generally happens when we do have too many parameters in a model, and we never know how many is too many.

But when we are doing LCAO-MO we know the relationship between a dependent variable (say, energy) and independent variables (LCAO-MO fficients). ANd we are certain that the bigger the number of AOs (and correspondingly of the coefficients), the better result could be expected with the the exact solution corresponding to using infinitely many AOs. So we should not worry about anything like overfitting. Yes, by increasing the number of AOs in a basis set we increase the number of variational parameters, but we know it can't do any harm: in the worst case¹ we will get the same energy as before the just another expansion of the basis.

This is similar to a Taylor series expansion of a function. The exact result corresponds to an infinite series, and by using a finite number of terms a function can only be approximated. But the more terms we use, the better approximation is. There is even a formal proof, but I think it should be intuitively clear that if an infinite series is the exact solution, than the later you truncate it the better approximation you get.

---

1) The worst case would be, for instance, when you expand a basis by including new functions that can be expressed as linear combinations of basis functions from the previously used basis set. You won't improve energy by doing so, but only increase the computational time. Note, that most of the programs have a simple defense mechanism against such situations: they are smart enough to detect the linear dependence in the basis and get rid of it prior to doing the calculations.

Answer 2

Perhaps we start with your picture first. It is indeed not applicable. Just imagine a vector $\vec{\psi}$ in a finite $n$-dimensional space and a set of other vectors $\vec{x}_i$. Your goal is to find the linear combination of $\vec{x}_i$ that is best describing your $\vec{\psi}$: $$ \vec{\psi} \approx \sum_i a_i \vec{x}_i $$ If you just think of it in the usual 3D euclidian vector space with arrows... it is intuitively obvious that using more basis vectors $\vec{x}_i$ gives a better approximation. In the case of finite $n$-dimensional spaces a set of $n$ linearly independent vectors gives even the exact linear combination. The beautiful thing about this picture is that the main task for ab initio programs in theoretical chemistry is searching for the best linear combination.

The only problem in praxis is that we have vector spaces with infinite dimensions and will never find a set that gives the exact linear combination. If we could, this would solve the Schrödinger equation.

If you look further on in text books, you are looking for the Hylleras-Undheim Theorem.

Please note that in the strict formulation this theorem is under the assumption of linear variational methods. In praxis it is even wider applicable.