Light travels at different speeds in different materials. The ratio of the speed at which light travels in a vacuum compared to the speed light travels in a given material is given by the index of refraction ($n$) of the material.
$${{n}=\mathrm{\frac{speed~ of~ light ~in~ vacuum}{speed ~of~ light~ in~ material}}}$$
The index of refraction is a basic property of a material. Further, the index of refraction of a material is wavelength dependent.
The beam of plane polarized light that we use in our polarimeter experiments is created by combining a beam of left circularly polarized light with a beam of right circularly polarized light. Those two beams interfere with each other to produce the plane polarized light we use in the polarimeter experiment. It is important to note that both the right and left circularly polarized beams are chiral as they trace out a right and left handed helix.
If the material in the polarimeter is not chiral, then the angle of the plane polarized light will not be rotated as it passes through the sample. However, if a chiral material is placed in the path of a plane polarized beam, then the left circularly polarized component of the beam will be rotated a different amount than the right circularly polarized component (the 2 beams are chiral and interact differently with the chiral sample) and we observe a rotation of our light beam as it passes through the polarimeter. In other words, the index of refraction of the left circularly polarized light ($n_L$) is different from the index of refraction of the right circularly polarized light ($n_R$) in chiral media.
Since $n$ itself is wavelength dependent, both $n_L$ and $n_R$ are also wavelength dependent and the observed rotation from our sample will change as we change the wavelength of our light beam.
This property of $n_L$ and $n_R$ is the basis of the Cotton effect. If you would like to read more about the Cotton effect and its use in chemistry see this earlier answer.