The primed version of $\mathrm{S_N1}$ and $\mathrm{S_N2}$ can only occur if there is a double bond in the vicinity of the leaving group as in your example. I have drawn both possibilities for the $\mathrm{S_N2}$ case in the scheme below.
Scheme 1: Comparison of the reaction products in an $\mathrm{S_N2}$ (top) and $\mathrm{S_N2'}$ (bottom) pathway.
The $\mathrm{S_N2}$ reaction is as you expect it to be. The $\mathrm{S_N2'}$ reaction uses the double bond as an electron relay system. Instead of the nucleophile (here: $\ce{OMe-}$) directly interacting with the $\unicode{x3c3}^*(\ce{C-Br})$ orbital, the π system interacts with the σ* orbital. The nucleophile then interacts with the π system’s LUMO (corresponding to the middle orbital of the allyl π system) to perform the attack. Since the nucleophilic attack and the leaving group are on different carbon atoms, relayed by the π system, this is not a direct $\mathrm{S_N2}$ but a derivative of it ($\mathrm{S_N2'}$).
The same logic can be applied to $\mathrm{S_N1'}$: here, the intermediate carbocation is not a localised one but an allyl cation.
The primed versions here are observed because not only the bimolecular attack on the highly substituted tertiary carbon ($\mathrm{S_N2}$) is unlikely — this carbon is tertiary which usually already implies no $\mathrm{S_N2}$ — but also the capturing of the carbocation under $\mathrm{S_N1}$ conditions is more likely to happen at the sterically much less hindered carbon atom.
As to why methanol predominantly attacks according to an $\mathrm{S_N1}$-type mechanism while methanolate predominantly follows an $\mathrm{S_N2}$-type mechanism I will refer you back to your textbook; the reasons are stated pretty often.
