From Roger E. Schirmer "Modern Methods of Pharmaceutical Analysis":
Solvents are generally ranked by polarity, but polarity is not a uniquely defined physical property of a substance. Hence the relative polarity of a solvent will be somewhat dependent on the method used to measure it....
Solvent polarity is a complex function of many parameter in addition to adsorption energy. A more recent ranking of solvents by Snyder is based on a combination of parameter such as dipole moment, proton acceptor or donor properties, and dispersion force solvent...
Snyder's polarity index ranks solvents according to a complex theoretical summation of these properties. As a rule, the higher the polarity index, the more polar the solvent.
Snyder's paper was published in Journal of Chromatography A, Volume 92, Issue 2, 22 May 1974, Pages 223-230 . Take a look at it to see how the index was calculated.
There are different types of polarity indices. And each of them has different parameters and ways to calculate the polarity.
Unlike polarity, dipole moment is a physical property. Polarity indices often take dipole moments as a parameters when calculating the polarity of solvents. The reason why there are polarity indices is because dipole moments alone couldn't explain the nature and interactions of solvents.
Part of your confusion stems from the repetition of the word "polar" to describe different phenomena. There is polarity of bonds and there is polarity of solvents. They are different things.