The instrument used is called a polarimeter. You basically shine monochromatic linear polarized light through a solution of known concentration and you measure the rotation of the light on the other end of a tube with defined length. You then normalize the rotation according to concentration and length of your tube to get the specific rotation which is then independent from concentration and length and only depends on temperature, solvent and wavelength of the light.
You can determine if it's rotated left or right by changing the concentration since the observed rotation is directly proportional to concentration. You also need to do this to determine if your rotation is more than a full 360°, since a measurement of 10° could also be 370° or 730° or 1090° or also −350°, −710°, and so on.
Let's look at this using a simple example and let us assume our tube has a length of 10 cm (which is the length used for the specific rotation, so we don't need to worry about this). Specific rotation is also normalized to a concentration of 1 g/100 mL.
To determine our specific rotation we could do 3 measurements at 1 g/100 mL, 2 g/100 mL and 0.5 g/100 mL.
- 1 g/100 mL : +160°
- 2 g/100 mL : −40°
- 0.5 g/100 mL : −100°
Now what does this tell us? If the specific rotation would be +160° then double the concentration should be 320° and half should be +80°, so that's clearly not the case. If it would be −200° instead of +160°, then double the concentration would be −400° which would show up as −40° and half the concentration would be −100°. So our specific rotation in this case is −200°.