I am studying sigmatropic reactions in context of pericyclic reactions and I am confused about the overall purpose of this type of reactions. I know that sometimes in sigmatropic reactions, more substituted alkene in the product in one reason why the reactions take place. However, in other types of sigmatropic reactions, like hydride shift, or when more substituted alkene product is not produced, I am having trouble seeing why this reactions occurs in the first place. Its almost like the electrons and double bonds are simple moving around to new locations, without any purpose.
Most of these reactions are indeed equilibrium reactions, meaning that they go forwards and backwards all the time and at a given point in time there will be a certain ratio of reactants to products corresponding to the equilibrium constant. Which side is favoured is often determined by minute stabilisations that are indeed very hard to rationalise.
Take for example the set of Wagner-Meerwein rearrangements that occur in the synthesis of cholesterol after squalene was cyclised. There appears to be no true stabilisation which would explain why one side is favoured over the other. In the end all we know is that the product is more stable or the reaction would not occur.