Reason for the generation of London Dispersion Forces

While talking about gaseous state of matter we came to the topic of London Dispersion Forces while are caused by the generation of dipole in one atom which induces a dipole in another. While talking about the cause of such I stated that since Schrodinger's equation for multi-electron atoms is time dependent and hence at some moment a dipole may get generated (due to the increase in the probability of electrons in that region). Though as for now I am thinking that what I said was quite vague and may not be the reason for it.

So what is the reason for generation of dipole moment in one atom to cause the London forces?

Let me crash the party here.

TL;DR: Dispersion interactions are not due to fluctuating induced-dipole attractions, even if everyone tries to explain it that way.

(I know that this must seem like an outrageous statement, and surely downvote fingers are itching now. Bear with me.)

Based on the Hellmann-Feynman theorem, it is known that the forces acting on a nucleus arise from two purely coulombic sources: Its attraction to its surrounding electron distribution, and its repulsion with other nuclei. Hence, the observation that e.g. the two atoms in a rare gas dimer are attracted to each other immediately implies that there is a concentration of electron density in between the nuclei, so that the resulting net forces pulls them "inwards" towards each other. As Feynman put it himself in 1939:

... the charge distribution of each is distorted from central symmetry, a dipole moment of order $$1/R^{7}$$ being induced in each atom. The negative charge distribution of each atom has its center of gravity moved slightly toward the other. It is not the interaction of these dipoles which leads to van der Waals' force, but rather the attraction of each nucleus for the distorted charge distribution of its own electrons that gives the attractive $$1/R^{7}$$ force.

This is the complete opposite picture of the induced-dipole explanation, where the electron densities "evade" each other by simultaneous displacements in the same direction.

In fact, Hunt has shown in a very laborious 1990 paper that Feynman's picture holds true not just for single atoms, but for any molecular geometry as well:

... Feynman's statement concerning forces between well-separated atoms in S states generalizes to interacting molecules A and B of arbitrary symmetry (with A and B in their ground electronic states). To leading order, the dispersion force on each nucleus I in molecule A results from the attraction of I to the dispersion-induced change in polarization of the electronic charge cloud on molecule A itself.

The Feynman explanation is not very popular, but there are in fact some authors who have picked it up and commented on it, most notably perhaps Richard Bader (of Atoms-In-Molecules fame). Politzer and Murray have written a nice article on the topic.

I know this is confusing at first, so I want to finish by mentioning a very intuitive way to rationalize this "inwards polarization" effect for a rare gas dimer. From another paper by Clark, Murray and Politzer:

What causes the polarization of the electronic densities toward each other (...)? This can be easily understood when it is noted that the electrostatic potential produced by the nucleus and electrons of any free ground-state atom is positive everywhere; the effect of the nucleus dominates over that of the dispersed electrons. This positive potential is what each atom “sees” of the other atom (...).

You are correct that simply stating that the Schrodinger equation is time dependent does not really answer the question. In fact, time-dependence is not a component of the derivation of the dispersion forces. At a surface level, they just represent how much the ground state energy of a system of polarizable molecules is lowered by their interactions. It's the same idea as any of the other intermolecular forces (coulomb, dipole-dipole, etc.) where some interaction between particles leads to a lower energy state than the free particles.

Generally, the London Dispersion interaction potential is not derived in a general chemistry class because it is purely quantum mechanical and rather involved. These slides provide a clear and thorough (albeit advanced) derivation, but I will explain the main points below.

Derivation Explanation

Let's imagine for this model that a polarizable atoms can be thought of as a simple harmonic oscillator with a positive charge ($$+q$$) on one end of a spring and a negative charge ($$-q$$) on the other end. The spring has an equilibrium length of 0, so that the atom has zero average dipole. This model of an polarizable atom captures the physics we want because applying a force of some kind to the spring will induce a dipole in this atom ($$\vec{p} = q \vec{d}$$).

So far we have only considered one of these atoms. What if we move to consider multiple of them together? Remember we're really interested in the energy difference between two non-interacting atoms and two interacting atoms, so let's start by finding the non-interacting ground state energy of two atoms. We'll let the two atoms have two different spring frequencies $$\omega_1$$ and $$\omega_2$$ respectively.

Setting $$z_1$$ and $$z_2$$ as the displacements of the springs from equilibrium for atoms 1 and 2 respectively and $$m$$ as the mass of an atom, we can write the total potential energy of the system as: $$U_{\text{non-interact}} = \frac{1}{2}\left(m \omega_1^2z_1^2 + m \omega_2^2z_2^2\right)$$

Solving the Schrodinger equation for this potential yields the ground state energy for the two non-interacting harmonic oscillators: $$E = \frac{\hbar}{2} (\omega_1 + \omega_2)$$

Now, we turn on the interaction and the problem becomes harder more interesting because the positions of the springs affect the magnitude of the interaction. Physically, because the dipole-dipole interaction can lower the energy, we expect that the interacting ground state of the two atoms will be a balance between minimizing the stretching of the springs (which increases the energy) and maximizing the amount of dipole-dipole interaction (which lowers the energy).

Setting $$z$$ as the distance between the two atoms, with some algebra one can show that the interaction energy between these two dipoles is given by $$U_{\text{interact}}(z,z_1,z_2) = \frac{q^2}{2\pi\epsilon_0}\frac{z_1 z_2}{z^3}$$

Now, our total potential energy for the system is given by: $$U_{\text{total}}(z,z_1,z_2) = U_{\text{non-interact}}(z_1,z_2) + U_{\text{interact}}(z,z_1,z_2)$$ While intimidating looking, this potential can also be solved exactly by changing variables to $$z_+ = (z_1 + z_2)$$ and $$z_- = (z_1 - z_2$$) and rearranging terms to make it look like the non-interacting potential energy. Once that is done the ground state energy can be found just as easily as in the non-interacting case.

With this done, we can find that the (one dimensional) energy stabilization is given by: $$U(z) = \frac{-\alpha^2 I}{2}\frac{q^2}{(4\pi\epsilon_0)^2}\frac{1}{z^6}$$ Where the final step was to replace some of our less measurable parameters with their atomic counterparts: polarizability ($$\alpha$$) and ionization potential ($$I$$)