Both first-order and second-order Jahn-Teller distortions play a very important role in chemistry.
It is often said that the Jahn-Teller effect is based on symmetry arguments, and hence nothing can be said about the extent or the exact form of the distortion.
What are these "symmetry arguments", and how can they be derived?
As already mentioned, the Jahn–Teller effect has its roots in group theory. The essence of the argument is that the energy of the compound is stabilised upon distortion to a lower-symmetry point group. This distortion may be considered to be a normal mode of vibration, with the corresponding vibrational coordinate $q$ labelling the "extent of distortion". There is one condition on the vibrational mode: it cannot transform as the totally symmetric irreducible representation of the molecular point group, as such a vibrational mode cannot bring about any distortion in the molecular geometry (it may lead to a change in equilibrium bond length, but not in the shape of the molecule). $\require{begingroup} \begingroup \newcommand{\En}[1]{E_n^{(#1)}} \newcommand{\ket}[1]{| #1 \rangle} \newcommand{\n}[1]{n^{(#1)}} \newcommand{\md}[0]{\mathrm{d}} \newcommand{\odiff}[2]{\frac{\md #1}{\md #2}}$
In the undistorted geometry (i.e. $q = 0$), the electronic Hamiltonian is denoted $H_0$. The corresponding unperturbed electronic wavefunction is $\ket{\n{0}}$, and the electronic energy is $\En{0}$. We therefore have
$$H_0 \ket{\n{0}} = \En{0}\ket{\n{0}} \tag{1}$$
Here, the Hamiltonian, wavefunction, and energy are all functions of $q$. We can expand them as Taylor series about $q = 0$:
$$\begin{align} H &= H_0 + q \left(\odiff{H}{q}\right) + \frac{q^2}{2}\left(\frac{\md^2 H}{\md q^2}\right) + \cdots \tag{2} \\ \ket{n} &= \ket{\n{0}} + q\ket{\n{1}} + \frac{q^2}{2}\ket{\n{2}} + \cdots \tag{3} \\ E_n &= \En{0} + q\En{1} + \frac{q^2}{2}\En{2} + \cdots \tag{4} \end{align}$$
In the new geometry (i.e. $q \neq 0$), the Schrodinger equation must still be obeyed and therefore
$$H\ket{n} = E_n \ket{n} \tag{5}$$
By substituting in equations $(2)$ through $(4)$ into equation $(5)$, one can compare coefficients of $q$ to reach the results:
$$\begin{align} \En{1} &= \left< \n{0} \middle| \odiff{H}{q} \middle| \n{0} \right> \tag{6} \\ \En{2} &= \left< \n{0} \middle| \frac{\md^2 H}{\md q^2} \middle| \n{0} \right> + 2\sum_{m \neq n}\frac{\left|\left<m^{(0)} \middle|(\md H/\md q)\middle|\n{0} \right>\right|^2}{\En{0} - E_m^{(0)}} \tag{7} \end{align}$$
The derivation of equations $(6)$ and $(7)$ will not be discussed further here.1
Distortions that arise due to the $\En{1}$ term are called first-order Jahn–Teller distortions, and distortions that arise from the $\En{2}$ term are called second-order Jahn–Teller distortions.
Recall that
$$E_n = \En{0} + \color{red}{q\En{1}} + \cdots \tag{8}$$
Therefore, if $\En{1} > 0$, then stabilisation may be attained with a negative value of $q$; if $\En{1} < 0$, then stabilisation may be attained with a positive value of $q$. These simply represent distortions in opposite directions along a vibrational coordinate. A well-known example is the distortion of octahedral $\ce{Cu^2+}$: there are two possible choices, one involving axial compression, and one involving axial elongation. These two distortions arise from movement along the same vibrational coordinate, except that one has $q > 0$ and the other has $q < 0$.
In order for there to be a first-order Jahn–Teller distortion, we therefore require that
$$\En{1} = \left<\n{0}|(\md H/\md q)| \n{0}\right> \neq 0 \tag{9}$$
Within group theory, the condition for the integral to be nonzero is that the integrand must contain a component that transforms as the totally symmetric irreducible representation (TSIR). Mathematically,
$$\Gamma_{\text{TSIR}} \in \Gamma_n \otimes \Gamma_{(\md H/\md q)} \otimes \Gamma_n \tag{10}$$
We can simplify this slightly by noting that the Hamiltonian, $H$, itself transforms as the TSIR. Therefore, $\md H/\md q$ transforms as $\Gamma_q$, and the requirement is that
$$\Gamma_{\text{TSIR}} \in \Gamma_n \otimes \Gamma_q \otimes \Gamma_n \tag{11}$$
In all point groups, for any non-degenerate irrep $\Gamma_n$, we find that $\Gamma_n \otimes \Gamma_n = \Gamma_{\text{TSIR}}$. Therefore, if $\Gamma_n$ is non-degenerate, then
$$\Gamma_n \otimes \Gamma_q \otimes \Gamma_n = \Gamma_q \neq \Gamma_{\text{TSIR}} \tag{12}$$
and the molecule is stable against a first-order Jahn–Teller distortion. Therefore, all closed-shell molecules ($\Gamma_n = \Gamma_{\text{TSIR}}$) do not undergo first-order Jahn–Teller distortions.
However, what will happen if $\Gamma_n$ is degenerate? Now, the product $\Gamma_n \otimes \Gamma_n$ contains other irreps apart from the TSIR.2 If the molecule possesses a vibrational mode that transforms as one of these irreps, then the direct product $\Gamma_n \otimes \Gamma_q \otimes \Gamma_n$ will contain the TSIR.
In a rather inelegant article,3 Hermann Jahn and Edward Teller worked out the direct products for every important point group and found that:
stability and degeneracy are not possible simultaneously unless the molecule is a linear one...
In other words, if a non-linear molecule has a degenerate ground state, then it is susceptible towards a (first-order) Jahn–Teller distortion.
Take, for example, octahedral $\ce{Cu^2+}$. This has a $\mathrm{^2E_g}$ term symbol (see this question) - which is doubly degenerate. The symmetric direct product $\mathrm{E_g \otimes E_g = A_{1g} \oplus E_g}$. Therefore, if we have a vibrational mode of symmetry $\mathrm{E_g}$, then distortion along this vibrational coordinate will occur to give a more stable compound. Recall that the vibrational mode cannot transform as the TSIR, so we can neglect the $\mathrm{A_{1g}}$ term.
What does an $\mathrm{e_g}$ vibrational mode look like? Here is a diagram:4
It's an axial elongation, which happens to match what we know of Cu(II). However, there is a catch. The vibrational mode is doubly degenerate (the other $\mathrm{e_g}$ mode is not shown), and any linear combination of these two degenerate vibrational modes also transforms as $\mathrm{e_g}$. Therefore, the exact form of the distortion can be any linear combination of these two degenerate modes. It can also involve negative coefficients, i.e. it might feature axial compression instead of elongation; there is no way to find that out using arguments purely based on symmetry.
Therefore, in this case, it is simply a coincidence that the $\mathrm{e_g}$ mode displayed is exactly the same as the form of the distortion seen in Cu(II). Nevertheless, it is encouraging to see that the axial elongation indeed transforms as $\mathrm{e_g}$ - which lends credence to the group theoretical analysis above.
On top of that, there's also no indication of how much distortion there is. That depends on (amongst other things) the value of $\En{1}$, and all we have said is that it is nonzero - we have not said how large it is.
This is what is meant by "impossible to predict the extent or the exact form of the distortion".
Pearson has written an article on second-order Jahn–Teller effects.5
For the second-order term, the energy correction is of the form
$$E_n = \En{0} + q\En{1} + \color{red}{\frac{q^2}{2}\En{2}} \cdots \tag{13}$$
Here, the $q^2$ term means that $\En{2}$ has to be negative if we want to see a second-order Jahn–Teller distortion. Unlike the first-order case, if $\En{2} > 0$, there will not be any distortion.
The second-order correction to the energy comprises two terms. The first term, $\left<\n{0}|(\md^2 H/\md q^2)|\n{0}\right>$, is always positive. (The proof of this is left to the reader. Hint: use the fact that $\md H/\md q$ is hermitian.) It may be interpreted as a restoring force that tries to bring the nuclei back to their original positions, and it is related to the fact that if the electronic state remains unperturbed (i.e. $\ket{n} = \ket{\n{0}}$), the unperturbed nuclear positions represent the most stable nuclear configuration.
The second term has the form
$$\sum_{m \neq n}\frac{\left|\left<m^{(0)} \middle|(\md H/\md q)\middle|\n{0} \right>\right|^2}{\En{0} - E_m^{(0)}} \tag{14}$$
which may seem like a slight monstrosity, but it is actually much easier to analyse than it looks. The summation over $m$ indicates that we are going to count every single electronic state $\ket{m}$ that is not the ground state $\ket{n}$. Since the numerator is a square modulus, it is either zero or positive, and since $\En{0} < E_m^{(0)}$, the denominator is always negative; therefore, this term is necessarily either zero or negative.
If $\En{2}$ is to be negative, then we need the second term to dominate the first. For this to occur, there are two prerequisites:
The denominator must be small, i.e. $\ket{m}$ must be a low-lying excited state such that the energy gap $\Delta E = \En{0} - E_m^{(0)}$ is small;
The numerator must not be zero, i.e. there must be a low-lying excited state of the appropriate symmetry $\ket{m}$ such that $\Gamma_m \otimes \Gamma_q \otimes \Gamma_n$ contains $\Gamma_{\text{TSIR}}$.
The first condition usually means that it will suffice to consider the first few excited states. In many of the examples I know of, the excited state that mixes with the ground state is the first excited state. In such cases one can even simply get rid of the sum and set $\ket{m}$ to be the first excited state.
These symmetry requirements are much less restrictive than previously, and second-order Jahn–Teller distortions tend to be much more widely seen than first-order distortions. A small selection of compounds in which second-order Jahn–Teller distortions are important are: p-block hydrides, $\ce{PbO}$, $\ce{Hg^2+}$, $\ce{WMe6}$, $\ce{R2Sn=SnR2}$, and anti-aromatic compounds such as cyclobutadiene. Let us use octahedral $\ce{Hg^2+}$ as an illustration.
In undistorted $O_\mathrm{h}$ symmetry, $\ce{Hg^2+}$ has a closed-shell $\mathrm{d^{10}}$ configuration and therefore its electronic ground state is $\Gamma_n = \mathrm{A_{1g}}$. However, upon excitation of one electron from the 5d orbitals (specifically the $\mathrm{e_g}$ set) to the 6s orbital (which transforms as $\mathrm{a_{1g}}$), the term symbol changes to
$$\Gamma_m = \mathrm{E_g \otimes A_{1g} = E_g} \tag{15}$$
Therefore, a vibrational mode transforming as $\mathrm{E_g}$ will facilitate a second-order distortion, since
$$\Gamma_m \otimes \Gamma_q \otimes \Gamma_n = \mathrm{E_g \otimes E_g \otimes A_{1g}} \tag{16}$$
contains the TSIR. Again, there is no way of knowing the exact form or the extent of the distortion; we only know that it transforms as $\mathrm{E_g}$. In the case of Hg(II), the distortion is manifested as an axial compression to give a "2 short, 4 long" coordination geometry, which is often described as "linear".
A second factor that favours the distortion is the extremely small 5d–6s gap in mercury, due to relativistic 5d destabilisation and 6s stabilisation. To see the importance of the small $\Delta E$, consider $\ce{Zn^2+}$, which has a larger 3d–4s gap; linear Zn(II) compounds are rare, while linear Hg(II) compounds are the norm.
Most of the time, the relevant excited state $\ket{m}$ arises from promotion of an electron from the HOMO to the LUMO. It is easy to show that if this is the case,
$$\Gamma_m \otimes \Gamma_n = \Gamma_{\text{HOMO}} \otimes \Gamma_{\text{LUMO}} \tag{17}$$
The second-order Jahn–Teller distortion can then be viewed as a reduction in symmetry, such that the HOMO and the LUMO, which transformed as different irreps in the undistorted geometry, now transform as the same irrep and therefore mix with each other. In the case of $\ce{Hg^2+}$:
This interpretation using the symmetry of individual orbitals, however, only works when the relevant excited state $\ket{m}$ is derived by excitation of an electron! In some (admittedly very rare) cases, it is possible that both $\ket{n}$ and $\ket{m}$ are derived from the same electronic configuration. This is the case for cyclobutadiene, and the Jahn–Teller effect in cyclobutadiene cannot be rationalised using orbital mixing. $\endgroup$
(1) For more details, look up perturbation theory in your quantum mechanics book of choice. In such treatments, the perturbation is usually formulated slightly differently: e.g. $H$ is taken as $H_0 + \lambda V$, and the eigenstates and eigenvalues are expanded as a power series instead of a Taylor series. Notwithstanding that, the principles remain the same.
(2) There is a subtlety in that the symmetric direct product must be taken. For example, in the $D_\mathrm{\infty h}$ point group, we have $\Pi \otimes \Pi = \Sigma^+ + [\Sigma^-] + \Delta$. The antisymmetric direct product $\Sigma^-$ has to be discarded.
(3) Jahn, H. A.; Teller, E. Stability of Polyatomic Molecules in Degenerate Electronic States. I. Orbital Degeneracy. Proc. R. Soc. A 1937, 161 (905), 220–235. DOI: 10.1098/rspa.1937.0142. n.b. Considering that I don't have a more elegant proof for it, I don't have much of a right to call it inelegant.
(4) Albright, T. A.; Burdett, J. K.; Whangbo, M.-H. Orbital Interactions in Chemistry, 2nd ed.; Wiley: Hoboken, NJ, 2013.
(5) Pearson, R. G. The second-order Jahn–Teller effect. J. Mol. Struct.: THEOCHEM 1983, 103, 25–34. DOI: 10.1016/0166-1280(83)85006-4.
