I upvoted the question because it is actually very insightful. We may think of simply removing the MgO layer, and voila, there is bare magnesium. I think not.
If you took bare Mg (no oxide at all) in a vacuum, and dropped powdered MgO onto the Mg, then exposed the metal to an argon atmosphere, I think you could blow off the oxide powder because there would be no significant adhesion.
In other words, the unreactivity of the oxide-coated magnesium arises because the oxide coat is strongly bonded to the metal. This transition from metal to metal oxide occurs over a few atoms and is probably very irregular. After all, even the surface of the metal won't even be flat to within 1 or 2 atoms.
Visible scratches on the metal after sandpapering prove that some mechanical effect has removed metal. If you remove metal, you must have broken some metal-metal bonds; and in the turmoil, at least some metal - oxygen bonds must also have been broken - even tho they might soon reform with O2 in air.
A similar, but chemical, film removal occurs when stainless steel, which is likewise protected by a very thin oxide coat, is treated with hydrochloric acid. The film is removed, and the steel corrodes. Because the reaction is so thorough, removing the whole oxide coat and a few atomic layers of metal as well, we don't stop to think about the transition layer. Different thicknesses can produce different colors as well as different degrees of protection.