Is density functional theory an ab initio method?

Question

The following comment by Wildcat made me think about whether density functional theory (DFT) can be considered an ab initio method.

@Martin-マーチン, this is sort of nitpicking, but DFT (where the last "T" comes from "Theory") can be considered as an ab-initio method since the theory itself is built from the first principles. The problem with the theory is that the exact functional is unknown, and as a result, in practice we do DFA calculations ("A" from "Approximation") with some approximate functional. It it DFA which is not an ab-initio method then, not DFT. :)

I always thought that ab initio refers to wave function based methods only. In principle the wave function is not necessary for the basis of DFT, but it was later introduced by Kohn and Sham for practical reasons.

The IUPAC goldbook offers a definition of ab initio quantum mechanical methods:

ab initio quantum mechanical methods
Synonym: non-empirical quantum mechanical methods
Methods of quantum mechanical calculations independent of any experiment other than the determination of fundamental constants. The methods are based on the use of the full Schroedinger equation to treat all the electrons of a chemical system. In practice, approximations are necessary to restrict the complexity of the electronic wavefunction and to make its calculation possible.

According to this, most density functional approximations (DFA) cannot be termed ab initio since almost all involve some empirical parameters and/or fitting. DFT on the other hand is independent of any of this. What I have my problems with is the second sentence. It states, that treatment of all electrons is necessary. This is technically not the case for DFT, because here only the electron density is treated. All electrons and the wavefunction are implicitly treated.

An earlier definition of ab initio can be found in Leland C. Allen and Arnold M. Karo, Rev. Mod. Phys., 1960, 32, 275.

By ab initio we imply: First, consideration of all the electrons simultaneously. Second, use of the exact nonrelativistic Hamiltonian (with fixed nuclei), $$\mathcal{H} = -\frac12\sum_i{\nabla_i}^2 - \sum_{i,a}\frac{Z_a}{\mathbf{r}_{ia}} + \sum_{i>j}\frac{1}{\mathbf{r}_{ij}} + \sum_{a,b}\frac{Z_aZ_b}{\mathbf{r}_{ab}}$$ the indices $i$, $j$ and $a$, $b$ refer, respectively, to the electrons and to the nuclei with nuclear charges $Z_a$, and $Z_b$. Third, an effort should have been made to evaluate all integrals rigorously. Thus, calculations are omitted in which the Mulliken integral approximations or electrostatic models have been used exclusively. These approximate schemes are valuable for many purposes, but present experience indicates that they are not sufficiently accurate to give consistent results in ab initio work.

This definition obviously does not include DFT, but this is probably due to the fact it was published before the Hohenberg-Kohn theorems. But in general this definition is still largely the same as in the goldbook.

Another point which confuses me are titles like:

"Potential Energy Surfaces of the Gas-Phase S_N2 Reactions $\ce{X- + CH3X ~$=$~ XCH3 + X-}$ $\ce{(X ~$=$~ F, Cl, Br, I)}$: A Comparative Study by Density Functional Theory and ab Initio Methods"
Liqun Deng , Vicenc Branchadell , Tom Ziegler, J. Am. Chem. Soc., 1994, 116 (23), 10645–10656.

And then again we have titles like:

"Ab Initio Density Functional Theory Study of the Structure and Vibrational Spectra of Cyclohexanone and its Isotopomers"
F. J. Devlin and P. J. Stephens, J. Phys. Chem. A, 1999, 103 (4), 527–538.

Unfortunately Koch and Holthausen, who wrote the probably most concise book on DFT, A Chemist's Guide to Density Functional Theory, never really refer to DFT as ab initio or clearly draw the line. The closest they come is on page 18:

In the context of traditional wave function based ab initio quantum chemistry a large variety of computational schemes to deal with the electron correlation problem has been devised during the years. Since we will meet some of these techniques in our forthcoming discussion on the applicability of density functional theory as compared to these conventional techniques, we now briefly mention (but do not explain) the most popular ones.

But that does not really answer my question. Throughout the book they use the term only in the form of conventional ab initio theory or in combination of explicitly stating wave function and variations thereof.

In my quite extensive research about DFT selection criteria I never came about the term 'ab initio DFT'.

So the question remains:
Is density functional theory an ab initio method?

Answer 1

First note that the acronym DFA I used in my comment originates from Axel D. Becke paper on 50 year anniversary of DFT in chemistry:

Let us introduce the acronym DFA at this point for “density-functional approximation.” If you attend DFT meetings, you will know that Mel Levy often needs to remind us that DFT is exact. The failures we report at meetings and in papers are not failures of DFT, but failures of DFAs. Axel D. Becke, J. Chem. Phys., 2014, 140, 18A301.

So, there are in fact two questions which must be addressed: "Is DFT ab initio?" and "Is DFA ab initio?" And in both cases the answer depend on the actual way ab initio is defined.

• If by ab initio one means a wave function based method that do not make any further approximations than HF and do not use any empirically fitted parameters, then clearly neither DFT nor DFA are ab initio methods since there is no wave function out there.
• But if by ab initio one means a method developed "from first principles", i.e. on the basis of a physical theory only without any additional input, then
  • DFT is ab initio;
  • DFA might or might not be ab initio (depending on the actual functional used).

Note that the usual scientific meaning of ab initio is in fact the second one; it just happened historically that in quantum chemistry the term ab initio was originally attached exclusively to Hartree–Fock based (i.e. wave function based) methods and then stuck with them. But the main point was to distinguish methods that are based solely on theory (termed "ab initio") and those that uses some empirically fitted parameters to simplify the treatment (termed "semi-empirical"). But this distinction was done before DFT even appeared.

So, the demarcation line between ab initio and not ab initio was drawn before DFT entered the scene, so that non-wave-function-based methods were not even considered. Consequently, there is no sense to question "Is DFT/DFA ab initio?" with this definition of ab initio historically limited to wave-function-based methods only. Today I think it is better to use the term ab initio in quantum chemistry in its more usual and more general scientific sense rather then continue to give it some special meaning which it happens to have just for historical reasons.

And if we stick to the second definition of ab initio then, as I already said, DFT is ab initio since nothing is used to formulate it except for the same physical theory used to formulate HF and post-HF methods (quantum mechanics). DFT is developed from the quantum mechanical description without any additional input: basically, DFT just reformulates the conventional quantum mechanical wave function description of a many-electron system in terms of the electron density.

But the situation with DFA is indeed a bit more involved. From the same viewpoint a DFA method with a functional which uses some experimental data in its construction is not ab initio. So, yes, DFA with B3LYP would not qualify as ab initio, since its parameters were fitted to a set of some experimentally measure quantities. However, a DFA method with a functional which does not involve any experimental data (except the values of fundamental constants) can be considered as ab initio method. Say, a DFA using some LDA functional constructed from a homogeneous electron gas model, is ab initio. It is by no means an exact method since it is based on a physically very crude approximation, but so does HF from the family of the wave function based methods. And if the later is considered to be ab initio despite the crudeness of the underlying approximation, why can't the former be also considered ab initio?

Answer 2

The convention used by many is that ab initio refers solely to wave-function based methods of various sorts and that first principles refers to either wave-function or DFT methods with little approximation.

I can't find a citation at the moment, but I know this convention is fairly widely used in, e.g., J. Phys. Chem. journals.

The IUPAC gold book doesn't have "first principles," but Google Scholar gives over 224,000 hits for "first principles DFT".

Answer 3

As already answered in the MatterModeling Stack Exchange, I think there is one aspect missing, and here I would like to quote my late PhD supervisor Jaap Snijders.

The most important aspect to know if a method is ab initio or not, is related to the integrals. If the integrals can be computed from the beginning, the method is ab initio; if not, then not. In DFT, DFAs and wavefunction methods, the integrals can be computed, and hence, these methods are ab initio.

This goes back to the Prologue of the book "Ab Initio Molecular Orbital Theory" by Hehre, Radom, v.R. Schleyer and Pople:

Soon after its formulation in 1925, it became clear that solution of the Schrödinger differential equation could, in principle, lead to direct quantitative prediction of most, if not all, chemical phenomena using only the values of a small of physical constants (Planck's constant, the velocity of light, and the masses and charges of electrons and nuclei). Such a procedure constitutes an ab initio approach to chemistry, independent of any experiment other than determination of these constants.

In semi-empirical methods (AM1, PM3, DFTB, xtb), some of the integrals are either estimated or approximated (from e.g. DFA results in case of DFTB/xtb, or from experimental data such as ionization potentials or other data), and therefore, these methods are not ab initio.