There are several arguments. Firstly, five-membered ring formation is generally kinetically more favourable than 6-membered ring formation. As ring size increases, the entropy of activation also increases – this is explained in greater detail in Clayden 2ed, pp 805–807. It is true that six-membered rings are less strained, but that is generally more of a thermodynamic consideration, and from a kinetic point of view the entropy factor happens to outweigh the enthalpy factor.
Secondly, for cationic ring-opening reactions (e.g. reactions of halonium ions, or epoxide opening under acidic conditions), it is usually the more substituted end that preferentially reacts. See e.g. Regioselectivity of bromination of alkenes. This is also explained in Clayden 2ed, pp 436–438.
However, probably the most relevant factor here is stereoelectronic in nature. Stereoelectronic effects refer to effects which depend on the spatial orientation ("stereo") of orbitals ("electronics"). In the case of an SN2 reaction, the key requirement is that the lone pair of the nucleophile (Nu) must be able to reach the σ* orbital of the C–X bond being broken. Since the σ* orbital points out of carbon diametrically opposite the leaving group X, this is equivalent to saying that we need a collinear Nu···C···X geometry in the transition state.
For ring-forming reactions, there are a set of guidelines – Baldwin's rules – which empirically describe whether these reactions are stereoelectronically favoured (i.e. whether the nucleophile and electrophile are capable of orienting themselves in a geometry that allows for reaction). In the case of six-membered ring formation via substitution at a tetrahedral carbon, the reactions are labelled 6-exo-tet or 6-endo-tet depending on whether the leaving group X is outside the 6-membered ring (exo), or inside the ring (endo). [Technically, 6-endo-tet reactions are not ring forming.]
It turns out that although 6-exo-tet reactions are perfectly OK, 6-endo-tet reactions simply do not happen, as the nucleophile cannot stretch itself to reach the C–X σ*.
The opening of a three-membered ring is somewhat intermediate between these two cases. Because the C–X bond is constrained in a small ring, the reaction leans towards being a disfavoured 6-endo-tet reaction, even though formally it is 6-exo (as the leaving group is outside the ring being formed):
The lack of overlap isn't obvious from the 2D diagrams here, but bear in mind that these compounds are not flat – especially in the case of the epoxide, where if the epoxide is in the plane of the paper, the alkyl substituent must be pointing up towards us (or down away from us).
As a result of this, ring opening of the iodonium ion to give a five-membered ring is much more favourable.
A similar example is seen in acid-catalysed epoxide opening.[1,2] The formation of the 6-membered ring (the tetrahydropyran) is not observed at all:
References
- Janda, K.; Shevlin, C.; Lerner, R. Antibody catalysis of a disfavored chemical transformation. Science 1993, 259 (5094), 490–493 DOI: 10.1126/science.8424171.
- Na, J.; Houk, K. N.; Shevlin, C. G.; Janda, K. D.; Lerner, R. A. The energetic advantage of 5-exo versus 6-endo epoxide openings: a preference overwhelmed by antibody catalysis. J. Am. Chem. Soc. 1993, 115 (18), 8453–8454. DOI: 10.1021/ja00071a067.
