Is it true that all of the physical laws and mathematics needed to describe chemistry are known?

Question

This question comes from a rather famous quote by Paul Dirac which goes like this,

The underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble. It therefore becomes desirable that approximate practical methods of applying quantum mechanics should be developed, which can lead to an explanation of the main features of complex atomic systems without too much computation. -Paul M. Dirac

I have seen this quote a number of times and it never struck me as being peculiar because to my knowledge the actual physical laws describing everything we care about in chemistry are known. However, I was recently at a talk (it was a presentation to one of my classes, but the speaker gave a talk about his research) about density functional theory, and this quote was shown as a sort of introduction for why density functional theory was developed and is applied as a form of approximation. After showing this quote, the speaker said "well that's not strictly speaking true..." and then moved on and didn't say anything else.

I meant to ask what he meant by this after the talk, but I forgot because I was asking about something else.

So, to expand on this:

Are there any physical laws which are unknown (or believed to be unknown) which directly relate to chemistry?

Are there any mathematics which have not been developed which are necessary to fully describe some aspect of known chemistry?

Answer 1

This sounds an awful lot like something I said at a talk this week, so I feel obligated to answer.

First, in terms of fundamental interactions, yes, excluding a quantum theory of gravity we have a quantum field theory for how the other fundamental forces work (electromagnetic/weak and strong).

@DavePhD mentions that Dirac was wrong at least up to the development of QED. This is true. Dirac could write the non-relativistic molecular Hamiltonian down. He knew that even if he couldn't solve it, all the physics were still there, and so in principle the system was "knowable." This is just like how we can't solve the gravitational many-body problem exactly but we definitely know how Newtonian gravity works.

Anyway, fast forward to today. We can write the QED equations of motion down which account for the time-evolution of all field operators. In principle this contains all the interactions necessary to describe molecules. Computational complexity aside, I think it is important to recognize that we must get rid of several degrees of freedom before we can even define a molecule. Most importantly we have to remove the possibility of electron-positron pair creation, since in QED particle number is not constant. Molecules, after all, are defined (in our chemists' minds) as having a fixed number of electrons. Heck, even "particles" themselves are not really present in QED, they are just excitations of the underlying quantum field.

We do something similar even in non-relativistic QM, where we fix the geometry of molecules via the Born Oppenheimer approximation (and treat nuclei as classical). If we didn't, the Schrodinger equation would describe every possible geometry of a collection of atoms and electrons (Well, geometry is not well defined in a electron-nuclear wave function, but you get the idea).

All this to say, writing the equations that govern a molecule will never be as simple as just "writing down the fundamental interactions", and I think Dirac got that wrong. Approximations will always be necessary as long as we hold onto a conception of a molecule as a fundamental object of study.

Molecular QED today is limited to effective Hamiltonian theories involving photon absorption and emission. But most of the same results can be derived from a classical view of EM fields.

Most of QED is unnecessary for an accurate description of chemistry. Pair creation processes are in energy ranges that we just don't access in the laboratory. The one area which it could excel is relativistic electronic structure theory. Our current attempts are based off of the Dirac equation, which really only holds for one spin-$1/2$ particle. Given the extension to multiple particles, we have to resort to approximate relativistic treatments. The most accurate -- in the sense that it contains the most physics -- relativistic two electron interaction term I've come across is the Breit interaction, but even that is an approximate electron-electron repulsion term. We don't know the exact relativistic term within the structure of relativistic electronic structure theory. But that's okay for now, since even including the Breit term is overkill for most molecular systems.

As far as not knowing all the fundamental interactions relevant for chemistry, let me finish with one example that I find particularly fascinating, and that is the nature of chirality of molecules. One area of study is whether or not one enantiomer of a chiral molecule is energetically more stable than the other. Even if there is a slight difference, over long periods of time it may explain why life tended to evolve using L-amino acids, among other things. This energy difference is hypothesized to be very small, on the order of 10$^{-11}$ J / mol.

Anyway, this theoretical difference in energy of chiral molecules cannot be explained using any theory of electromagnetic interactions, because the EM interactions are identical in chiral molecules. Instead, the difference in energy (if there is one) must come from a parity-violating term, which shows up only in the electroweak interaction. So this area of study is known as electroweak chemistry. As far as I know, the exact form of this parity violating term is up for debate (and there may be multiple terms), since it necessarily has to couple to some sort of magnetic perturbation, like spin-orbit coupling. Because no one really knows exactly what this term looks like, it makes it difficult for theorists to predict what the possible energy difference between enantiomers should be. Which then makes it very difficult for spectroscopists to know what to probe for.

So this is an example of a fundamental interaction relevant to chemistry (albeit a small one), that we don't really know much about, but would give huge insights into the evolution of life as we know it.

Answer 2

First, it is one thing to know the basis for a "law", and quite another to mathematically calculate the effects of that law. Consider just carbon, for example... chains may be made of thousands of atoms, with various functional groups attached to each. Though, as Dirac stated, it helps to have "shortcuts" to computation such as Fast Fourier transforms, there are still problems that cannot be solved in a "reasonable" time.

Second, if there are unknown laws, how would we know about it (not to quote Rumsfeld on unknown unknowns)?

And finally, even if all physical laws were known and understood, it would still be impossible to predict everything: Kurt Gödel's incompleteness theorems show that in a complex system (it does not have to be very complex; basic grammar-school mathematics qualifies), questions can be asked that cannot be proved true or false. This extends to chemistry and physics.

Answer 3

Dirac is probably right but, even if he isn't, it probably isn't important for chemistry.

The issue highlighted by Dirac is that, even if we do understand all the relevant laws of quantum mechanics as far as they determine chemical properties, that doesn't help us turn chemistry into a branch of mathematics. The problem is that while we understand the equations we don't have good ways of solving those equations except for the simples systems. For example, we only have exact solutions to the electron wave function equations (which is what determines most chemical properties) for the simplest possible atoms (one nucleus, one electron). Everything else is an approximation.

This should not be a surprise. The three body problem for Newtonian gravity has no exact solutions (or, more strictly, only a very small number for some very special cases). Quantum wave functions are more complex than that and systems with multiple electrons are not going to have neat mathematical solutions.

What this implies is that we can't reliably predict the chemical properties of anything more complicated than hydrogen atoms from the physical laws they obey even if we completely understand the laws. We can approximate but it is hard to tell if reality deviates from the rules because our approximations are poor or because we don't understand some detail of the rules.

So even if there is some subtle detail of the laws we don't understand, it would be hard to verify the implications for chemistry.

There might be a few areas where obscure parts of quantum mechanics do impact chemistry (though speculation is all we have right now because of the limitation described above). Normally in the quantum mechanics used for chemistry, we are just looking at electromagnetic forces, and that is complicated enough. Some people have speculated that other forces might have small influences that matter for chemistry. For example, there is speculation that some interaction with nuclear forces might explain life's preference for single optical isomers in many living structures. The speculation suggests that optical isomers have very slightly different energies because of a tiny interaction with the asymmetry in non-electromagnetic forces (see this example). But these effects are, if they exist, small compared to the uncertainty in our predictions based on the well-known laws.

So the dominant problem is chemistry is the quality of our approximations not the potential existence of entirely new laws.

Answer 4

As already pointed out in a comment by John Custer, Dirac was 100% correct in

The underlying physical laws necessary for the mathematical theory of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble. It therefore becomes desirable that approximate practical methods of applying quantum mechanics should be developed, which can lead to an explanation of the main features of complex atomic systems without too much computation.

For light systems, typically thought as H-Kr, non-relativistic quantum mechanics i.e. the Schrödinger equation is sufficient to describe chemistry; for heavier nuclei you need to use relativistic quantum mechanics i.e. the Dirac equation which is a bit more complicated. In many cases we can invoke the Born-Oppenheimer method, and assume that the nuclei are moving in the instantaneous field of the electrons; now all we need to solve is the electronic problem.

We know that the exact solution to the electronic problem in this case is achievable with the full configuration interaction (FCI) a.k.a. exact diagonalization method, in which you describe the electronic many-particle wave function by a weighted sum of electronic configurations a.k.a. Slater determinants as $|\Psi \rangle = \sum_k c_k |\Phi_k\rangle$. These electronic configurations are built by distributing $N$ electrons into $K$ single-particle states a.k.a. orbitals. For the method to be accurate, $K\gg N$, and actually to get the exact solution you need $K \to \infty$.

Now, to find the ground state (as well as any excited states), you just need to diagonalize the many-electron Hamiltonian in the basis of the electron configurations. But, the problem is that the number of the electron configurations grows extremely rapidly.

If we assume that we are looking at a spin singlet state, then you have $N/2$ spin-up and $N/2$ spin-down electrons. For each spin, there are ${K \choose N/2} = \frac {K!} {\frac N 2 ! (K-\frac N 2)!} $ ways to populate the orbitals. The total number of electron configurations for $N$ electrons in $K$ orbitals, typically denoted as ($N$e,$K$o), is then ${K \choose N/2}^2 = \left[ \frac {K!} {\frac N 2 ! (K-\frac N 2)!} \right]^2$.

Even for the case of a very small number of orbitals, $K=N$, the number of configurations quickly becomes huge. (8e,8o) has 4900 configurations, (10e,10o) has 63 504, (12e,12o) has 853 776, (14e,14o) has 11 778 624, (16e,16o) has 165 636 900, (18e,18o) has 2 363 904 400, (20e,20o) has 34 134 779 536, and (22e,22o) has 497 634 306 624.

Although you can still solve the (8e,8o) problem with dense matrix algebra on modern computers, you see that very quickly you have to become very smart in how to diagonalize the matrix. Because the Hamiltonian is a two-particle operator, it is extremely sparse in the basis of the electron configurations: if two configurations differ by the occupation of more than two orbitals, the Hamiltonian matrix element is zero by Slater and Condon's rules. Moreover, for large problem sizes you also want to avoid storing the matrix, which is why you want to use an iterative method. (The famous Davidson method for iterative diagonalization was actually developed exactly for this purpose!)

With smart algorithms, billion-configuration calculations i.e. the (18e,18o) problem have been possible since the early 1990s, see e.g. Chem. Phys. Lett. 169, 463 (1990). However, despite the huge increase in computational power in the last 30 years, the barrier has barely budged: as far as I am aware, the largest FCI problem that has been solved is the (22e,22o) calculation in J. Chem. Phys. 147, 184111 (2017).

The thing to note here is that even the (22e,22o) calculation is not large enough to solve a single atom in an exact fashion: you need a lot more orbitals to achieve quantitative accuracy with experiment. Although a high-lying orbital contributes only very little to the correlation energy, there are A LOT of them.

Exactly as Dirac wrote, approximations are needed. Density-functional approximations are extremely popular in applications, but they are far from exact. On the other hand, high-accuracy studies often employ the coupled-cluster method, which is a reparametrization of the FCI method; however, it, too would exhibit exponential scaling were it not truncated - i.e. approximated.

Answer 5

Dirac published that statement in Quantum Mechanics of Many Electron Systems, received 12 March 1929.

In 1948 Verwey and Overbeek demonstrated experimentally that the London dispersion interaction is even weaker than 1/$r^6$ at long distance (say hundreds of Angstroms or more).

Casimir and Polder soon thereafter explained with quantum electrodynamics (QED) that the dependence should be 1/$r^7$ at relatively long distances.

See section 3.1.1 Van der Waals Forces, in Some Vacuum QED Effects

So Dirac was wrong at least up to the development of QED.

Answer 6

The answer to both of your questions is yes. There are unknown physical laws that are directly related to chemistry and there are mathematics that have not yet been developed and are necessary to fully describe some aspect of known chemistry.

But before I expand on my answer, let me add that Dirac wrote what you quote right after he developed the first version of quantum electrodynamics in 1927. Around 50 years later, he changed his mind, admitting that quantum electrodynamics was an incomplete theory and he invited his colleagues to "seek a new relativistic quantum mechanics".

There are several aspects of known chemistry that cannot be explained by current physical laws. I will mention three.

Emergence

Although we knew the fundamental physical laws that apply to matter, this does not mean that the rest of the disciplines are only applications of those laws. As Anderson wrote in his classic "More is different" [1], biology is not applied chemistry, nor is chemistry applied physics. Emergent properties are those that only exist at a certain level of organization of matter and cannot be derived from lower levels.

Structure

Quantum mechanics is incompatible with molecular structure, as pointed out in several articles by Woolley. As explained in [2], the Schrödinger equation cannot differentiate between the following three molecules

Attempts to explain molecular structure introduce generalizations of the Schrödinger equation.

Reactions

The incompatibility between chemical reactions and quantum mechanics was highlighted by Prigogine [3]:

Quantum mechanics, in its orthodox form, corresponds to a deterministic time-reversible description. This is not so for chemistry. Chemical reactions correspond to irreversible processes creating entropy. That is, of course, a very basic aspect of chemistry, which shows that it is not reducible to classical dynamics or quantum mechanics.

He and his group developed extensions of quantum mechanics for irreversible processes.

References

1. More Is Different: Broken symmetry and the nature of the hierarchical structure of science. 1972: Science 177(4047) 393–396. Anderson, P. W.
2. Must a molecule have shape? 1988: New Scientist 120, 53–57. Woolley, Richard Guy.
3. Chemical Kinetics and Dynamics 2003: Annals of the New York Academy of Sciences 988, 128–132. Prigogine, Ilya.