1
$\begingroup$

I am doing experiments with the QM9 dataset from

and began with some simple checks on results from the literature. Consistent with what the authors of the paper

  • C.R. Collins, G.J. Gordon, O.A. Von Lilienfeld and D.J. Yaron, Constant size descriptors for accurate machine learning models of molecular properties. J. Chem. Phys. 148 (2018), 241718.

report, I find 3993 small QM9 molecules with 7 or fewer heavy atoms, which serves as their training set. But I cannot reproduce the Null row of their Table II, probably because I don't understand well enough what they actually do there. In Section V.A they say:

A “null” model is used to provide a baseline measure of the difficulty of the prediction task. The null model always predicts the mean value of the training data.

They don't specify precisely which quantity they predict, but on p.6 they report values in kcal/mol, and in Table II they talk about atomization energies, so I suppose the target is one of the energies provided by QM9. But there energies are in Hartree and hence must be converted to kcal/mol.

Using the atomization energy (QM9 property 13), I get 319.48 Ha for the mean atomization energy on the training set. With this mean as constant null predictor I obtain an MAE of 30.84 Ha = 19,353 kcal/mol on the training set, and of 31,319, 48,632, and 58,000 kcal/mol, on the first 20,000, 50,000, and 133,000 QM9 molecules, respectively. (The data used are in https://arnold-neumaier.at/nullQM9.txt. Calculating the mean and the MAEs for the mean is a standard triviality.) But these numbers are over 100 times larger than the numbers stated in the Null row of their Table II. (The table is also on p.10 of the arXiv preprint https://arxiv.org/abs/1701.06649 - Section V.A is there called 5.1.)

Without conversion, the numbers would be far too small. The other properties from QM9 don't fare better.

Perhaps I misunderstood what they meant by the Null model, or there is an undocumented preprocessing step for getting the target energies for prediction?

Could anyone please resolve this discrepancy?

$\endgroup$
10
  • 2
    $\begingroup$ This question is missing the crucial part: What exactly are you doing to get to your numbers? Apart from this, this appears to be more a question about the computations or calculations and less about the chemistry or even the database. It's probably better suited at Matter Modeling. I'll close the question to resolve these issues first. $\endgroup$ Dec 10, 2022 at 9:07
  • 1
    $\begingroup$ I think the edit has not made the question any clearer. How do you calculate the energies? I unfortunately don't have access to the publication you cite, so I can't look at the context. The first reply from Matter Modeling concurs with me, that data like an input file would be very helpful. I'm still hesitant to send this question over. $\endgroup$ Dec 11, 2022 at 20:15
  • $\begingroup$ I used the energies tabulated in the QM9 dataset as Property 13. All I did was calculating the mean over the training set specified in the paper, and the mean absolute deviation of it (i) on the training set, and (ii) on the three initial subsets of the database for which values were given in Table II. The computation of these is trivial (and I had it checked by someone else), but I can add programs if really wanted. $\endgroup$ Dec 11, 2022 at 20:43
  • $\begingroup$ @Martin: I also linked to the arXiv preprint of the paper. $\endgroup$ Dec 11, 2022 at 20:48
  • 1
    $\begingroup$ I agree with @Martin-マーチン that this is better suited at Matter Modeling. It's a newer site, so yes there are fewer "QM9" questions, but there are far more ML experts. $\endgroup$ Dec 12, 2022 at 16:09

1 Answer 1

2
$\begingroup$

First off, let me stress that it's important to have a null model or baseline model for comparison in machine learning. In other words, does the ML model presented in the paper do better than a simple alternative.

Atomization energies are calculated from the electronic structure calculation of the molecule and the relevant atoms. In other words, how much energy does it take to dissociate the molecule into the constituent atoms (e.g., 6 carbon and 6 hydrogen atoms for benzene). Consequently, atomization energies depend on the number and type of atoms in the molecule. The atomization energy for benzene is higher than that of methane, because you're breaking more bonds.

My reading of this paper, was that the null model consisted of the average energies for each atom, e.g. $E_{C_6H_6} = 6E_C + 6E_H$ and $E_{C_5NH_5} = 5E_C + E_N + 5E_H$ with the various $E_C$ for each element fit from least squares. (Certainly that's what I'd consider as a null model for this, because standard atomization energies use the isolated atoms, so least-squares fit would at least adjust for atoms in a molecule.)

Your comment suggests that least-squares null model performs better than the reported null model in the paper.

In that case, I don't know, but if/when I see David Yaron next, I'll ask.

$\endgroup$
2
  • $\begingroup$ Incidentally, I'd strongly suggest avoiding the QM9. I know it has become a standard set, but the molecules are really strange compared to "typical molecules" from PubChem or similar sets. $\endgroup$ Dec 19, 2022 at 20:24
  • 1
    $\begingroup$ The least squares appropach is not their null model but their model 1 - though they use ridge regression, hence get slightly different values. The null model must have gotten the $E_H$, etc from an unspecified, different source. $\endgroup$ Dec 20, 2022 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.