7
$\begingroup$

I am trying to compare the performance of few Quantum Chemistry property prediction ML models. I was looking at the following table from DOI: 10.1039/c7sc02664a

enter image description here

The problem is that it does not contain any units. A preprint (https://arxiv.org/pdf/1909.00259.pdf, Table 4) suggests that the energy units are Hartree. Among energy properties, HOMO, LUMO, and gap have reasonable values for Hartree unit. However, values reported for U0, U, H, and G look absurd. As a comparison, look at MAE for mean value of each property:

enter image description here

These are in [eV] units (1 Ha = 27.212 eV). In other words, for instance MPNN models MAE for U property is 55.78 [eV] vs 8.25 [eV] for the mean baseline. This does not make sense at all. [k cal/mol] sounds more reasonable but then why half the energies in one unit the other half in other unit, and then why there are no units in a paper published by Royal Society of Chemistry. Seems like I'm missing something. Thanks!

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
7
$\begingroup$

Yes, you're exactly right - multiple papers in chemistry ML drop the units.

There are even comparisons (usually by statistics, ML or comp. sci. researchers) where models are compared by "averaging" errors down a column like that. Of course that's meaningless, since you can't average electron volts or Hartree (energies), Debye (dipole moments), and volume (polarizabilities).

Worse, energies are often computed as atomization energies - so for large molecules, they can be enormous...

In my opinion, a more relevant and meaningful statistic would be the Mean Absolute Percent Error (MAPE) which is unitless and easier to understand (e.g., 1% error? 0.01% error?)

That said, the original QM9 paper gives the units Table 3: Scientific Data (2014) 1, art. 140022 QM9 data table

Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set. $\endgroup$
    – Geoff Hutchison
    Sep 26, 2019 at 18:04
  • $\begingroup$ Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out. $\endgroup$
    – Blade
    Sep 26, 2019 at 19:20
  • $\begingroup$ Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv). $\endgroup$
    – Blade
    Sep 26, 2019 at 20:02
  • 2
    $\begingroup$ I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol $\endgroup$
    – Geoff Hutchison
    Sep 26, 2019 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.