In phosphorus’ case, the connection between stability of the allotrope and its colour is only accidental.
White phosphorus, i.e. $\ce{P4}$ molecules, has very strained bonds because of the very small tetrahedral angles the atoms need to accomodate for. Going by the edges of a regular tetrahedron, the bond angles are $60^\circ$ which is very strained. This is the reason for its reactivity. Its colour is white, because there are no electron transitions excited by visible light, much like similar compounds of other elements.
Black phosphorus, on the other hand, forms large extensive networks which are well on the way to being metallic. Metallic bonding is typically such a widespread bonding network that every atom contributes to every molecular orbital with more or less the same contribution. You can compare this to large organic π systems. This allotrope is more stable because each phosphorus atom has bond angles that well suit the corresponding angles between atomic orbitals (somewhere just above $90^\circ$) and because that large network is hard to break. And since we have a lot of molecular orbitals with rather small differences in energy, light of the entire visible spectrum can be absorbed. There will always be a corresponding transition. Because of this, black phosphorus appears black.
Red phosphorus is somewhere in-between. The network is not as extensive, it is not as ordered and thus there are not as many molecular orbitals with good energy differences. Therefore, not all visible light is absorbed but green light is preferentially absorbed (giving a red colour). And also therefore, the reactivity is somewhere in-between white and black phosphorus.
(There is also violet phosphorus, which is somewhere between red and black.)