I am doing experiments with the QM9 dataset from
and began with some simple checks on results from the literature. Consistent with what the authors of the paper
- C.R. Collins, G.J. Gordon, O.A. Von Lilienfeld and D.J. Yaron, Constant size descriptors for accurate machine learning models of molecular properties. J. Chem. Phys. 148 (2018), 241718.
report, I find 3993 small QM9 molecules with 7 or fewer heavy atoms, which serves as their training set. But I cannot reproduce the Null row of their Table II, probably because I don't understand well enough what they actually do there. In Section V.A they say:
A “null” model is used to provide a baseline measure of the difficulty of the prediction task. The null model always predicts the mean value of the training data.
They don't specify precisely which quantity they predict, but on p.6 they report values in kcal/mol, and in Table II they talk about atomization energies, so I suppose the target is one of the energies provided by QM9. But there energies are in Hartree and hence must be converted to kcal/mol.
Using the atomization energy (QM9 property 13), I get 319.48 Ha for the mean atomization energy on the training set. With this mean as constant null predictor I obtain an MAE of 30.84 Ha = 19,353 kcal/mol on the training set, and of 31,319, 48,632, and 58,000 kcal/mol, on the first 20,000, 50,000, and 133,000 QM9 molecules, respectively. (The data used are in https://arnold-neumaier.at/nullQM9.txt. Calculating the mean and the MAEs for the mean is a standard triviality.) But these numbers are over 100 times larger than the numbers stated in the Null row of their Table II. (The table is also on p.10 of the arXiv preprint https://arxiv.org/abs/1701.06649 - Section V.A is there called 5.1.)
Without conversion, the numbers would be far too small. The other properties from QM9 don't fare better.
Perhaps I misunderstood what they meant by the Null model, or there is an undocumented preprocessing step for getting the target energies for prediction?
Could anyone please resolve this discrepancy?