Why does a methyl shift occur during the pinacol-pinacolone rearrangement?

Question

Why does a methyl shift occur during the pinacol-pinacolone rearrangement?

I was told that methyl shift is done to stabilise the carbocation but here we are shifting the carbocation from secondary to primary.

Answer 1

but here we are shifting the carbocation from secondary to primary

Firstly the shift is not from secondary to primary but from tertiary to secondary.

I was told that methyl shift is done to stabilise the carbocation

Yes, a methyl shift is done to stabilise a carbocation. And it is done in this case for the same reason, to increase the relative stability.

Now you might think, "How can a secondary carbocation be more stable than a tertiary carbocation? Even after having a highly electronegative atom next to it. "

Throughout all this thinking you are forgetting the fact that a carbocation is an electron deficient species and an oxygen atom has some lone pairs.

When an carbocation is formed next to an oxygen, the oxygen atom donates its lone pair via the +R effect which is very much stronger than its -I effect.

This fact can also be verified from this article^[1]:

Because heteroatoms such as oxygen and nitrogen are more electronegative than carbon, you might expect that they would by definition be electron withdrawing groups that destabilize carbocations. In fact, the opposite is often true: if the oxygen or nitrogen atom is in the correct position, the overall effect is carbocation stabilization. This is due to the fact that although these heteroatoms are electron withdrawing groups by induction, they are electron donating groups by resonance, and it is this resonance effect which is more powerful. (We previously encountered this same idea when considering the relative acidity and basicity of phenols and aromatic amines in section 7.4). Consider the two pairs of carbocation species below: In the more stable carbocations, the heteroatom acts as an electron donating group by resonance: in effect, the lone pair on the heteroatom is available to delocalize the positive charge. In the less stable carbocations the positively-charged carbon is more than one bond away from the heteroatom, and thus no resonance effects are possible. In fact, in these carbocation species the heteroatoms actually destabilize the positive charge, because they are electron withdrawing by induction.

Reference:

[1]: Carbocation Structure and Stability https://chem.libretexts.org/@go/page/31459 (accessed Jun 4, 2021).