Why do post-Hartree-Fock methods fail to predict the direction of the dipole moment of carbon monoxide?

Question

In carbon monoxide the dipole moment (negative to positive) points towards the oxygen, as I explained it in How can the dipole moment of carbon monoxide be rationalised by molecular orbital theory?

A calculation using density functional approximation, the level of theory is BP86/def2-QZVPP, yields the correct direction: \begin{align} \ce{{}^{\ominus}\!:C#O:^{\oplus}} && \text{Dipole:}~|\mathbf{q}|=0.19~\mathrm{D} && \text{Direction:}~\longrightarrow \end{align}

 ---------------------------------------------------------------------
 Center     Atomic      Atomic             Coordinates (Angstroms)
 Number     Number       Type             X           Y           Z
 ---------------------------------------------------------------------
      1          6           0        0.000000    0.000000   -0.648987
      2          8           0        0.000000    0.000000    0.486740
 ---------------------------------------------------------------------
[...]
 Dipole moment (field-independent basis, Debye):
    X=              0.0000    Y=              0.0000    Z=              0.1907  Tot=              0.1907

Using Møller-Plesset Perturbation Theory of second order, i.e. MP2/def2-QZVPP, we obtain \begin{align} \ce{{}^{\ominus}\!:C#O:^{\oplus}} && \text{Dipole:}~|\mathbf{q}|=0.30~\mathrm{D} && \text{Direction:}~\longleftarrow \end{align}

 ---------------------------------------------------------------------
 Center     Atomic      Atomic             Coordinates (Angstroms)
 Number     Number       Type             X           Y           Z
 ---------------------------------------------------------------------
      1          6           0        0.000000    0.000000   -0.648450
      2          8           0        0.000000    0.000000    0.486337
 ---------------------------------------------------------------------
[...]
 Dipole moment (field-independent basis, Debye):
    X=              0.0000    Y=              0.0000    Z=             -0.3002  Tot=              0.3002

There is no improvement going to other post-Hartree_Fock methods, as the following table indicates. A positive value means that the dipole is directed towards the carbon; the used basis set is in all cases def2-QZVPP. \begin{array}{lr} \text{Method} & \mathbf{q}(\overrightarrow{\ce{CO}})\\\hline \text{Experimental}^1 & 0.11\\\hline \text{HF} & -0.14\\\hline \text{MP2} & -0.30\\ \text{MP3} & -0.22\\ \text{MP4(SDQ)} & -0.27\\ \text{MP4(SDTQ)//MP4(SDQ)} & -0.27\\ \text{CCSD} & -0.25\\ \text{CCSD(T)//CCSD} & -0.25\\ \text{CISD} & -0.22\\ \text{QCISD} & -0.26\\ \hline \text{BP86} & 0.19\\ \text{PBE0} & 0.12\\ \text{M11} & 0.06\\ \text{B3LYP} & 0.10\\ \hline \text{B2PLYP} & -0.07\\ \hline \end{array}

This failure probably already starts with Hartree-Fock. Where does the deficiency lie in the HF description in this particular case? And why is this deficiency not corrected by post-HF methods? Even the double hybrid functional B2PLYP cannot predict the direction of the dipole moment correctly.

---

1. According to Gernot Frenking, Christoph Loschen, Andreas Krapp, Stefan Fau, and Steven H. Strauss, J. Comp. Chem., 2007, 28 (1), 117-126. the value can be found in J. S. Muenter. J. Mol. Spectrosc. 1975, 55, 490. (I don't have access to that publication.)

---

Sample Input for Gaussian 09

#p MP2/def2QZVPP
scf(tight) opt(verytight,maxcycle=100)
symmetry(loose)
gfinput gfoldprint iop(6/7=3)

carbon monoxide

0 1
C   0.0 0.0 1.135
O   0.0 0.0 0.0

Answer 1

The question itself is void, as all methods with the exception of Hartree-Fock predict the direction of the dipole moment correctly. Deathbreath found a Full CI calculation in Jeremy P. Coe, Daniel J. Taylor, and Martin J. Paterson. J. Comp. Chem. 2013, 34 (13), 1083-1093 of which luckily an open version exists (arXiv:1301.4904 [physics.chem-ph]). The following passage is taken from the latter:

The groundstate dipole moment of carbon monoxide, although fairly small, when calculated using HF strikingly has the incorrect sign compared with experimental results. Previous work has suggested that the accuracy of the dipole calculation depends on the amount of correlation accounted for.¹⁴ The bond length (2.1316 Bohr) and the experimental dipole value (0.122 Debye) are taken from Ref. 15. The positive value for the dipole here signifies a polarity of $\ce{C^−O^+}$.
With a cc-pVDZ basis, two frozen core orbitals and a cut-off value of $c_\mathrm{min} = 5 \times 10^{−3}$, we see in Fig. 1 [omitted] that the MCCI method, starting from close to the incorrectly signed result of the HF single SD, quickly reaches a correctly signed value which converges at around half of the FCI value.

---

¹⁴ G. E. Scuseria, M. D. Miller, F. Jensen, and J. Geertsen. J. Chem. Phys. 1991, 94, 6660.
¹⁵ J. S. Muenter. J. Mol. Spectrosc. 1975, 55, 490.

The problem indeed is just a technical one, resulting from analysing the wrong density. Gaussian 09 by default only analyses the SCF density (see population, density). In order to obtain values for the post-HF methods, the density=current keyword needs to be specified. The updated table therefore becomes: \begin{array}{lr} \text{Method} & \mathbf{q}(\overrightarrow{\ce{CO}})/\mathrm{D}& \mathbf{d}(\ce{CO})/\mathrm{10^{-10}m}\\\hline \text{Experimental}^{15} & 0.122 & 1.128\\\hline \text{HF} & -0.14 & 1.102\\\hline \text{MP2} & 0.26 & 1.135\\ \text{MP3} & 0.08 & 1.118\\ \text{MP4(SDQ)} & 0.11 & 1.118\\ \text{MP4(SDTQ)//MP4(SDQ)} & \text{n.a.} \\ \text{CCSD} & 0.08 & 1.124\\ \text{CCSD(T)//CCSD} & \text{n.a.}\\ \text{CISD} & 0.05 & 1.118\\ \text{QCISD} & 0.08 & 1.127\\ \hline \text{BP86} & 0.19 & 1.135\\ \text{PBE0} & 0.12 & 1.122\\ \text{M11} & 0.06 & 1.119\\ \text{B3LYP} & 0.10 & 1.124\\ \hline \text{B2PLYP} & 0.15 & 1.129\\ \hline \end{array}

---

Sample Input for Gaussian 09 for analysis
(requires the addition of %chk=mp2qzvpp.chk to the sample input of the question)

%oldchk=mp2qzvpp.chk
#p MP2/def2QZVPP
Geom=check
Guess=(Read,Only)
Density=(Check, Current)

dens-curr

0 1

---

I apologise for getting everyone worked up over a stupid beginner mistake.