Why does the energy gap for π - π* transitions shrink with the size of the pi-conjugated system?

Quoting from this site:

As conjugated pi systems become larger, the energy gap for a π - π* transition becomes increasingly narrow, and the wavelength of light absorbed correspondingly becomes longer. The absorbance due to the π - π* transition in 1,3,5-hexatriene, for example, occurs at 258 nm, corresponding to a ΔE of 111 kcal/mol.

It's also quite easy to see that this is true glancing at various dye absorption tables.

But why is this true? I can understand why the optical cross-section would increase, implying a higher quantum efficiency at some maximum absorbance wavelength (is this generally true?), but why would the energy gap between the ground state and the excited state correspondingly shrink?

-
For those interested in the matter: please also see this follow-up question. –  Philipp Mar 5 at 13:32

Klaus Warzecha's answer pretty much answers your question. But I know that this subject is easier to understand if supported by some pictures. That's why I will take the same route as Klaus at explaining the concept behind why the absorption in conjugated systems is shifted to higher wavelengths but I will provide some pictures on the way.

In a conjugated carbon chain or ring system you can think of the $\ce{C}$ atoms as $\text{sp}^{2}$-hybridized. So, each carbon has 3 $\text{sp}^{2}$ orbitals which it uses to form $\sigma$ bonds and 1 $\text{p}$ orbital which is used to form $\pi$ bonds. It is the $\text{p}$ orbitals that are responsible for the conjugation and their combinations according to the LCAO model are the interesting part since the HOMO and LUMO of the system will be among the molecular orbitals formed from the conjugated $\text{p}$ orbitals.

For a start take ethene, the simplest $\pi$-system, being comprised of only 2 carbon atoms. When you combine two atomic orbitals you get two molecular orbitals. These result from combining the $\text{p}$ orbitals either in-phase or out-of-phase. The in-phase combination is lower in energy than the original $\text{p}$ orbitals and the out-of-phase combination is higher in energy than the original $\text{p}$ orbitals. The in-phase combination accounts for the bonding molecular orbital ($\pi$), whilst the out-of-phase combination accounts for the antibonding molecular orbital ($\pi^{*}$).

Now, what happens when you lengthen the conjugated system by combining two ethene fragments? You get to butadiene. Butadiene has two $\pi$ bonds and so four electrons in the $\pi$ system. Which molecular orbitals are these electrons in? Since each molecular orbital can hold two electrons, only the two molecular orbitals lowest in energy are filled. Let's have a closer look at these orbitals. In $\Psi_1$, the lowest-energy bonding orbital, the electrons are spread out over all four carbon atoms (above and below the plane) in one continuous orbital. There is bonding between all the atoms. The other two electrons are in $\Psi_2$. This orbital has bonding interactions between carbon atoms 1 and 2, and also between 3 and 4 but an antibonding interaction between carbons 2 and 3. Overall, in both the occupied $\pi$ orbitals there are electrons between carbons 1 and 2 and between 3 and 4, but the antibonding interaction between carbons 2 and 3 in $\Psi_2$ partially cancels out the bonding interaction in $\Psi_1$. This explains why all the bonds in butadiene are not the same and why the middle bond is more like a single bond while the end bonds are double bonds. If we look closely at the coefficients on each atom in orbitals $\Psi_1$ and $\Psi_2$, it can be seen that the bonding interaction between the central carbon atoms in $\Psi_1$ is greater than the antibonding one in $\Psi_2$. Thus butadiene does have some double bond character between carbons 2 and 3, which explains why there is the slight barrier to rotation about this bond.

You can construct the molecular orbitals of butadiene by combining the molecular orbitals of the two ethene fragments in-phase and out-of-phase.

This method of construction also shows why the HOMO-LUMO gap of butadiene is smaller than that of ethene. The molecular orbital $\Psi_2$, which is the HOMO of butadiene, is the out-of-phase combination of two ethene $\pi$ orbitals, which are the HOMO of ethene. Thus, the HOMO of butadiene is higher in energy than the HOMO of ethene. Furthermore, the molecular orbital $\Psi_3$, which is the LUMO of butadiene, is the in-phase combination of two ethene $\pi^{*}$ orbitals, which are the LUMO of ethene. Thus, the LUMO of butadiene is higher in energy than the LUMO of ethene. It follows that the HOMO-LUMO energy gap is smaller in butadiene than in ethene and thus butadiene absorbs light with longer wavelenghts than ethene.

If you continue to lengthen the $\pi$ system by adding more ethene fragments you will see that the HOMO and LUMO are getting closer and closer together the longer the $\pi$ system becomes.

-
This is a terrific answer! –  user4717 Mar 4 at 18:20
@user4717 Thanks, it's definitely a rather lengthy one :) –  Philipp Mar 4 at 18:23
Looks like it took some work... so can we interpret jumping the gap from one particular HOMO to one particular LUMO configuration an example of promoting the dye to a particular singlet excitation state? –  user4717 Mar 4 at 18:25
@user4717 Yes, promoting an electron from a filled to an empty molecular orbital will lead to an excited state. Another thing that the MO picture conveys very nicely is that such a promotion would lead to a change in the bonding situation in the molecule. For example, if you promote an electron from $\Psi_2$ to $\Psi_3$ the middle bond between carbons 2 and 3 is strengthened while the outer bonds between carbons 1 and 2 and carbons 3 and 4 are weakened. You can draw a lot of useful conclusions from MO pictures. –  Philipp Mar 4 at 18:35
That makes a lot of sense. So if we look here: files.chem.vt.edu/chem-ed/quantum/jablonsk.html, $S_1$, $S_2$, ... correspond to events where an electron is promoted from a HOMO to a LUMO (the exact jump depending on the particular electronic configuration), and the intermediate eigenstates correspond to different vibrational/etc. modes? –  user4717 Mar 4 at 18:40

Let's have an extreme example of 9 conjugated double bonds. That makes your $\pi$ HOMO orbital. The LUMO, $\pi^\star$, has one node, the middle double bond being antibonding, so it will consist of two times 4 conjugated $\pi$ bonds. To say, the difference in energy between HOMO and LUMO is quite small, therefore the excitation energy is low.

-

The effect is already "visible" on the comparison of ethene ($\ce{H2C=CH2}$) and 1,3-butadiene ($\ce{H2C=CH-CH=CH2}$).

Let's build butadiene by combination of the molecular orbitals (MOs) of of ethene. Combination of the $\pi$ orbitals gives one new occupied MO without a nodal plane (this is lower in energy than the $\pi$ orbital of ethene) and another occupied one with one nodal plane. This is our HOMO (highest occupied molecular orbital) of butadiene.

The HOMO of butadiene has a higher energy than that of ethene.

Now let's have look at the antibonding $\pi^*$ orbitals of ethene and their combination. Again, we obtain two new MOs, one higher in energy than the $\pi^*$ of ethene and another lying lower. This is our new LUMO (lowest unoccupied molecular orbital.

The LUMO of butadiene has a lower energy than that of ethene.

As a result, the HOMO-LUMO gap is lower, and the absorption shifted to higher wavelength.

-
While you certainly show that the energy gap closes with increasing size of the $\pi$ system, I'm not sure it appropriately addresses the why part of the original question. –  tschoppi Mar 4 at 15:27
@tschoppi Let's wait and see :) In the worst case, you'll have to give an in-depth lecture on HMO theory ;) –  Klaus Warzecha Mar 4 at 15:54