See the footnotes I've included if you would like to see more of the detail behind a specific statement.
The figure below compares the reflectance spectrum for silver and gold (let's forget about aluminum, it's not relevant to this discussion; also keep in mind that where reflectance is low, absorbance is high and vice-versa).

The absorption (reduced reflectance) seen at lower wavelengths by silver and gold is caused by an electron transition (caused by the absorption of light) between their $\mathrm{4d \rightarrow 5s}$ and $\mathrm{5d \rightarrow 6s}$ orbitals respectively. Silver absorbs (reflects less) in the ultraviolet, no visible light is appreciably absorbed. Therefore, if we shine white light on silver, none of the visible wavelengths are absorbed, they are all reflected and silver appears as a shiny, white metallic substance. In fact, jewelers say that silver is the whitest of the metals. On the other hand, gold absorbs (reflects less) blue light in the visible region. When white light shines on gold, the blue light is absorbed and yellow light is reflected (see this article on complementary colors for more detail on how removing certain colors from white light produces different colors). So our question becomes, why is gold's $\mathrm{5d \rightarrow 6s}$ electronic transition shifted into the visible region?
$\mathrm{s}$-Electrons are strongly attracted to the nucleus because their probability density is higher near the nucleus than other types of orbitals ($\mathrm{p, d, f,}$ etc.). Simple Bohr theory tells us that these strongly attracted $\mathrm{s}$ electrons in atoms with a large number of protons (heavy nuclei) would have to travel at a considerable fraction of the speed of light to avoid "falling into" the nucleus (1). Special relativity takes things a step further and explains that, for these heavy nuclei with their $\mathrm{s}$ electrons traveling over half the speed of light, the electron mass will increase and as mass increases (2), orbital radius decreases (3).
Further, as the orbital radius decreases and the electron is closer to the nucleus it becomes more stabilized and the energy of the orbital decreases. As the various $\mathrm{s}$ electrons (and to a lesser degree $\mathrm{p}$ electrons, but $\mathrm{d}$ and $\mathrm{f}$ electrons don't have enough electron density close to the nucleus so they are not directly affected by these relativistic effects and their orbits are not contracted) move closer to the nucleus, the other non-$\mathrm{s}$ (or $\mathrm{p}$) electrons are better screened from the nucleus. So although the various $\mathrm{s}$ orbitals are contracted and placed at lower energy, the various $\mathrm{d}$ and $\mathrm{f}$ orbital radii shift further out and occur at higher energy due to this increased screening effect.
Hence, in the gold atom, we would expect the $\mathrm{5d}$ orbital to be raised in energy (screening effect) and the $\mathrm{6s}$ orbital to be lowered in energy (relativistic orbital contraction effect). The net effect is a smaller energy separation between these two orbitals and a red-shift in the $\mathrm{5d \rightarrow 6s}$ absorption to the lower energy (compared to the absorption of silver which is not subject to the same magnitude of relativistic effects) blue region of the spectrum, resulting in a yellowish color for light reflected from gold.
(1) The ratio of the speed of an electron traveling in the first Bohr orbit to the speed of light is given by the handy equation:
$$v_\mathrm{rel} = \frac{Z}{137}$$
where $Z$ is the atomic number of the element under consideration and $137$ is the speed of light in atomic units, also known as the fine structure constant. Consequently, a $\mathrm{1s}$ electron in silver $(Z=47)$ will travel around $\frac{47}{137} = 34\%$ the speed of light, while a similar electron in gold $(Z=79)$ will travel at $58\%$ the speed of light.
(2) The relativistic mass of an electron is given by:
$$m_\mathrm{rel} = \frac{m_\mathrm{e}}{\sqrt{1-\left(\frac{v_\mathrm{rel}}{c}\right)^2}}$$
where $m_\mathrm{e}$, $v_\mathrm{rel}$, and $c$ are the electron rest mass, the velocity of the electron, and the vacuum speed of light, respectively. The following figure provides a graphical representation of the result.

(3) The following equation relates the ratio of the relativistic radius of the first Bohr orbit $R_\mathrm{rel}$ to the normal radius $R_\circ$, to the relativistic velocity of the electron:
$$\frac{R_\mathrm{rel}}{R_\circ} = \sqrt{1-\left(\frac{v_\mathrm{rel}}{c}\right)^2}$$
As the relativistic velocity of the electron increases, the orbital radius contracts (the above ratio becomes smaller). For silver, the first Bohr radius contracts ~6%, while for gold the contraction is ~18%.