I understand that nucleophiles typically attack the electrophilic carbonyl carbon of the lactone and open the ring through a nucleophilic acyl substitution. Has there ever been the case where the nucleophiles open the lactone ring by attacking the carbon adjacent to the oxygen (not the carbonyl oxygen) via a SN2 reaction instead? Is there a preferred pathway for certain nucleophiles to take and why so?
Yes, it is possible. If you consider the chemistry of a four-membered lactone (a β-lactone), it is able to act as an alkylating agent, as nucleophiles can attack the C–O bond in an SN2 fashion.
One example is in the synthesis of a series of β-substituted amino acids (J. Am. Chem. Soc. 1985, 107 (24), 7105–7109). An intramolecular Mitsunobu reaction of Cbz-protected serine 1 forms the β-lactone 2 which is then opened by a variety of nucleophiles. Here, benzylthiolate is used as an example, giving N-Cbz-S-benzylcysteine 3 as the product:
The authors note that:
Only relatively “hard” nucleophiles like ammonia and methoxide attack the carbonyl to give acyl-oxygen cleavage. In earlier reports serine β-lactones were used mainly for synthesis of seryl peptides by attack of amines at the carbonyl. However, with ammonia as the nucleophile, simply altering the solvent [from MeCN to THF] inverts the 3:1 amide/amine ratio.
The following article has a wide selection of more examples (see references 6–8): J. Org. Chem. 2002, 67 (14), 4680–4683.
The reactivity you describe also occurs in α-lactones. The most common example is the diazotization of amino acids, followed by nucleophilic ring opening (e.g. Org. Synth. 1988, 66, 151):
The reaction proceeds with overall retention of configuration, via an initial SN2 displacement of N2 by the carboxylic acid, followed by a second SN2 ring-opening at the α-carbon by the nucleophile (here chloride).
See also Organic Chemistry (2nd Ed), Clayden et al., p 875.