This question comes from a rather famous quote by Paul Dirac which goes like this,

The underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble. It therefore becomes desirable that approximate practical methods of applying quantum mechanics should be developed, which can lead to an explanation of the main features of complex atomic systems without too much computation. -Paul M. Dirac

I have seen this quote a number of times and it never struck me as being peculiar because to my knowledge the actual physical laws describing everything we care about in chemistry are known. However, I was recently at a talk (it was a presentation to one of my classes, but the speaker gave a talk about his research) about density functional theory, and this quote was shown as a sort of introduction for why density functional theory was developed and is applied as a form of approximation. After showing this quote, the speaker said "well that's not strictly speaking true..." and then moved on and didn't say anything else.

I meant to ask what he meant by this after the talk, but I forgot because I was asking about something else.

So, to expand on this:

Are there any physical laws which are unknown (or believed to be unknown) which directly relate to chemistry?

Are there any mathematics which have not been developed which are necessary to fully describe some aspect of known chemistry?

  • 11
    $\begingroup$ The wave function of a multi-electron atom can not be solved for. That is a pretty basic one. $\endgroup$ – Jon Custer May 5 '16 at 2:33
  • 1
    $\begingroup$ Electronegativity still doesn't have a solid mathematical description. Electron correlation still has no exact mathematical description. $\endgroup$ – CoffeeIsLife May 5 '16 at 3:11
  • $\begingroup$ Need to upvote Jon's comment like 500 more times. $\endgroup$ – Lighthart May 5 '16 at 4:48
  • $\begingroup$ A relevant link from another physicist's point of view. Follow the links inside for more. $\endgroup$ – Nicolau Saker Neto May 5 '16 at 10:18

This sounds an awful lot like something I said at a talk this week, so I feel obligated to answer.

First, in terms of fundamental interactions, yes, excluding a quantum theory of gravity we have a quantum field theory for how the other fundamental forces work (electromagnetic/weak and strong).

@DavePhD mentions that Dirac was wrong at least up to the development of QED. This is true. Dirac could write the non-relativistic molecular Hamiltonian down. He knew that even if he couldn't solve it, all the physics were still there, and so in principle the system was "knowable." This is just like how we can't solve the gravitational many-body problem exactly but we definitely know how Newtonian gravity works.

Anyway, fast forward to today. We can write the QED equations of motion down which account for the time-evolution of all field operators. In principle this contains all the interactions necessary to describe molecules. Computational complexity aside, I think it is important to recognize that we must get rid of several degrees of freedom before we can even define a molecule. Most importantly we have to remove the possibility of electron-positron pair creation, since in QED particle number is not constant. Molecules, after all, are defined (in our chemists' minds) as having a fixed number of electrons. Heck, even "particles" themselves are not really present in QED, they are just excitations of the underlying quantum field.

We do something similar even in non-relativistic QM, where we fix the geometry of molecules via the Born Oppenheimer approximation (and treat nuclei as classical). If we didn't, the Schrodinger equation would describe every possible geometry of a collection of atoms and electrons (Well, geometry is not well defined in a electron-nuclear wave function, but you get the idea).

All this to say, writing the equations that govern a molecule will never be as simple as just "writing down the fundamental interactions", and I think Dirac got that wrong. Approximations will always be necessary as long as we hold onto a conception of a molecule as a fundamental object of study.

Molecular QED today is limited to effective Hamiltonian theories involving photon absorption and emission. But most of the same results can be derived from a classical view of EM fields.

Most of QED is unnecessary for an accurate description of chemistry. Pair creation processes are in energy ranges that we just don't access in the laboratory. The one area which it could excel is relativistic electronic structure theory. Our current attempts are based off of the Dirac equation, which really only holds for one spin-$1/2$ particle. Given the extension to multiple particles, we have to resort to approximate relativistic treatments. The most accurate -- in the sense that it contains the most physics -- relativistic two electron interaction term I've come across is the Breit interaction, but even that is an approximate electron-electron repulsion term. We don't know the exact relativistic term within the structure of relativistic electronic structure theory. But that's okay for now, since even including the Breit term is overkill for most molecular systems.

As far as not knowing all the fundamental interactions relevant for chemistry, let me finish with one example that I find particularly fascinating, and that is the nature of chirality of molecules. One area of study is whether or not one enantiomer of a chiral molecule is energetically more stable than the other. Even if there is a slight difference, over long periods of time it may explain why life tended to evolve using L-amino acids, among other things. This energy difference is hypothesized to be very small, on the order of 10$^{-11}$ J / mol.

Anyway, this theoretical difference in energy of chiral molecules cannot be explained using any theory of electromagnetic interactions, because the EM interactions are identical in chiral molecules. Instead, the difference in energy (if there is one) must come from a parity-violating term, which shows up only in the electroweak interaction. So this area of study is known as electroweak chemistry. As far as I know, the exact form of this parity violating term is up for debate (and there may be multiple terms), since it necessarily has to couple to some sort of magnetic perturbation, like spin-orbit coupling. Because no one really knows exactly what this term looks like, it makes it difficult for theorists to predict what the possible energy difference between enantiomers should be. Which then makes it very difficult for spectroscopists to know what to probe for.

So this is an example of a fundamental interaction relevant to chemistry (albeit a small one), that we don't really know much about, but would give huge insights into the evolution of life as we know it.

| improve this answer | |
  • $\begingroup$ Ha. You probably couldn't have answered in that much detail if I asked after the talk. The part about electroweak interactions is particularly interesting. And thanks for the talk! $\endgroup$ – jheindel May 8 '16 at 18:26

First, it is one thing to know the basis for a "law", and quite another to mathematically calculate the effects of that law. Consider just carbon, for example... chains may be made of thousands of atoms, with various functional groups attached to each. Though, as Dirac stated, it helps to have "shortcuts" to computation such as Fast Fourier transforms, there are still problems that cannot be solved in a "reasonable" time.

Second, if there are unknown laws, how would we know about it (not to quote Rumsfeld on unknown unknowns)?

And finally, even if all physical laws were known and understood, it would still be impossible to predict everything: Kurt Gödel's incompleteness theorems show that in a complex system (it does not have to be very complex; basic grammar-school mathematics qualifies), questions can be asked that cannot be proved true or false. This extends to chemistry and physics.

| improve this answer | |
  • 2
    $\begingroup$ Only very recently has mathematical undecidability been rigorously determined for a "reasonable" physical problem. Even then, there was a lot of discussion about the interpretation of the results, which I am nowhere near qualified to relay. $\endgroup$ – Nicolau Saker Neto May 5 '16 at 10:24

Dirac is probably right but, even if he isn't, it probably isn't important for chemistry.

The issue highlighted by Dirac is that, even if we do understand all the relevant laws of quantum mechanics as far as they determine chemical properties, that doesn't help us turn chemistry into a branch of mathematics. The problem is that while we understand the equations we don't have good ways of solving those equations except for the simples systems. For example, we only have exact solutions to the electron wave function equations (which is what determines most chemical properties) for the simplest possible atoms (one nucleus, one electron). Everything else is an approximation.

This should not be a surprise. The three body problem for Newtonian gravity has no exact solutions (or, more strictly, only a very small number for some very special cases). Quantum wave functions are more complex than that and systems with multiple electrons are not going to have neat mathematical solutions.

What this implies is that we can't reliably predict the chemical properties of anything more complicated than hydrogen atoms from the physical laws they obey even if we completely understand the laws. We can approximate but it is hard to tell if reality deviates from the rules because our approximations are poor or because we don't understand some detail of the rules.

So even if there is some subtle detail of the laws we don't understand, it would be hard to verify the implications for chemistry.

There might be a few areas where obscure parts of quantum mechanics do impact chemistry (though speculation is all we have right now because of the limitation described above). Normally in the quantum mechanics used for chemistry, we are just looking at electromagnetic forces, and that is complicated enough. Some people have speculated that other forces might have small influences that matter for chemistry. For example, there is speculation that some interaction with nuclear forces might explain life's preference for single optical isomers in many living structures. The speculation suggests that optical isomers have very slightly different energies because of a tiny interaction with the asymmetry in non-electromagnetic forces (see this example). But these effects are, if they exist, small compared to the uncertainty in our predictions based on the well-known laws.

So the dominant problem is chemistry is the quality of our approximations not the potential existence of entirely new laws.

| improve this answer | |
  • $\begingroup$ I think that the importance of "analytic solutions" as a measuring stick of whether a system is understood or not is overblown, because what makes a solution "analytic" is arbitrarily defined. It sounds much like separating numbers between constructible and non-constructible by virtue of the fact that the former can be produced with a compass and straightedge, while the latter can't. Just an artefact from old Greek geometers. $\endgroup$ – Nicolau Saker Neto May 5 '16 at 10:41
  • $\begingroup$ Non-relativistic quantum mechanics and relativistic Dirac equation have finite accuracy. It means if the error in approximation is smaller than the intrinsic error of the underlying equations, we are still fine. $\endgroup$ – Rodriguez May 5 '16 at 10:45
  • 1
    $\begingroup$ @NicolauSakerNeto My distinction isn't about the difference between "analytic" and approximate solutions. It is about the difficulty of even getting good approximate solutions, which is still large. That's why computational chemistry needs lots of computer power. $\endgroup$ – matt_black May 5 '16 at 11:05

Dirac published that statement in Quantum Mechanics of Many Electron Systems, received 12 March 1929.

In 1948 Verwey and Overbeek demonstrated experimentally that the London dispersion interaction is even weaker than 1/$r^6$ at long distance (say hundreds of Angstroms or more).

Casimir and Polder soon thereafter explained with quantum electrodynamics (QED) that the dependence should be 1/$r^7$ at relatively long distances.

See section 3.1.1 Van der Waals Forces, in Some Vacuum QED Effects

So Dirac was wrong at least up to the development of QED.

| improve this answer | |

As already pointed out in a comment by John Custer, Dirac was 100% correct in

The underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble. It therefore becomes desirable that approximate practical methods of applying quantum mechanics should be developed, which can lead to an explanation of the main features of complex atomic systems without too much computation.

For light systems, typically thought as H-Kr, non-relativistic quantum mechanics i.e. the Schrödinger equation is sufficient to describe chemistry; for heavier nuclei you need to use relativistic quantum mechanics i.e. the Dirac equation which is a bit more complicated. In many cases we can invoke the Born-Oppenheimer method, and assume that the nuclei are moving in the instantaneous field of the electrons; now all we need to solve is the electronic problem.

We know that the exact solution to the electronic problem in this case is achievable with the full configuration interaction (FCI) a.k.a. exact diagonalization method, in which you describe the electronic many-particle wave function by a weighted sum of electronic configurations a.k.a. Slater determinants as $|\Psi \rangle = \sum_k c_k |\Phi_k\rangle$. These electronic configurations are built by distributing $N$ electrons into $K$ single-particle states a.k.a. orbitals. For the method to be accurate, $K\gg N$, and actually to get the exact solution you need $K \to \infty$.

Now, to find the ground state (as well as any excited states), you just need to diagonalize the many-electron Hamiltonian in the basis of the electron configurations. But, the problem is that the number of the electron configurations grows extremely rapidly.

If we assume that we are looking at a spin singlet state, then you have $N/2$ spin-up and $N/2$ spin-down electrons. For each spin, there are ${K \choose N/2} = \frac {K!} {\frac N 2 ! (K-\frac N 2)!} $ ways to populate the orbitals. The total number of electron configurations for $N$ electrons in $K$ orbitals, typically denoted as ($N$e,$K$o), is then ${K \choose N/2}^2 = \left[ \frac {K!} {\frac N 2 ! (K-\frac N 2)!} \right]^2$.

Even for the case of a very small number of orbitals, $K=N$, the number of configurations quickly becomes huge. (8e,8o) has 4900 configurations, (10e,10o) has 63 504, (12e,12o) has 853 776, (14e,14o) has 11 778 624, (16e,16o) has 165 636 900, (18e,18o) has 2 363 904 400, (20e,20o) has 34 134 779 536, and (22e,22o) has 497 634 306 624.

Although you can still solve the (8e,8o) problem with dense matrix algebra on modern computers, you see that very quickly you have to become very smart in how to diagonalize the matrix. Because the Hamiltonian is a two-particle operator, it is extremely sparse in the basis of the electron configurations: if two configurations differ by the occupation of more than two orbitals, the Hamiltonian matrix element is zero by Slater and Condon's rules. Moreover, for large problem sizes you also want to avoid storing the matrix, which is why you want to use an iterative method. (The famous Davidson method for iterative diagonalization was actually developed exactly for this purpose!)

With smart algorithms, billion-configuration calculations i.e. the (18e,18o) problem have been possible since the early 1990s, see e.g. Chem. Phys. Lett. 169, 463 (1990). However, despite the huge increase in computational power in the last 30 years, the barrier has barely budged: as far as I am aware, the largest FCI problem that has been solved is the (22e,22o) calculation in J. Chem. Phys. 147, 184111 (2017).

The thing to note here is that even the (22e,22o) calculation is not large enough to solve a single atom in an exact fashion: you need a lot more orbitals to achieve quantitative accuracy with experiment. Although a high-lying orbital contributes only very little to the correlation energy, there are A LOT of them.

Exactly as Dirac wrote, approximations are needed. Density-functional approximations are extremely popular in applications, but they are far from exact. On the other hand, high-accuracy studies often employ the coupled-cluster method, which is a reparametrization of the FCI method; however, it, too would exhibit exponential scaling were it not truncated - i.e. approximated.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.