It is the same reasoning that is used to predict relative barrier heights in amines.
In the example pictured below, the imine starts with the nitrogen lone pair in "more or less" an $\ce{sp^2}$ orbital. In the transition state for interconversion, the lone pair is in a $\ce{p}$ orbital.
The term "more or less" is important. If we start with a molecule that has the nitrogen lone pair in an orbital that has a little bit more p character than $\ce{sp^2}$ (say $\ce{sp^{2.2}}$ for example), then it is already closer to the $\ce{p}$ hybridized transition state and the activation energy required to reach the transition state will be reduced. On the other hand, If we start with a molecule that has the nitrogen lone pair in an orbital that has a little bit less p character (more $\ce{s}$ character) than $\ce{sp^2}$ (say $\ce{sp^{1.8}}$ for example), then it is further from the $\ce{p}$ hybridized transition state and the activation energy required to reach the transition state will be greater.
In the case of oximes ($\ce{X=OH}$) or other molecules where the substituent on nitrogen is electronegative (electron withdrawing), the electron density in the $\ce{N-X}$ bond is shifted towards the electronegative atom. Consequently there is less electron density in the nitrogen portion of this bond. As a result the nitrogen rehybridizes. Since $\ce{s}$ orbitals are lower in energy than $\ce{p}$ orbitals, orbitals with increased $\ce{s}$ character are better able to stabilize electrons. Since there is less electron density in the orbital nitrogen is using for this bond, $\ce{s}$ character is removed from the orbital. This "extra" $\ce{s}$ character is used to stabilize the orbital containing the lone pair of electrons. The lone pair orbital now has more $\ce{s}$ character and is further away from the $\ce{p}$ hybridized transition state - the barrier to reach this transition state increases. If our substituent is an alkyl group (X=R, an imine), the carbon is less electronegative then nitrogen, there will be increased electron density in the nitrogen orbital used to form the $\ce{N-R}$ bond. The nitrogen will hybridize such that there is more $\ce{s}$ character in this orbital so as to better stabilize these electrons. As a consequence, there will less $\ce{s}$ character (more $\ce{p}$ character) in the lone pair orbital. The lone pair orbital is therefore closer to the $\ce{p}$ hybridized transition state and less energy will be required to reach the transition state.
As a result of this reasoning $\ce{NF3}$ has a higher inversion barrier then $\ce{NH3}$ and oximes have a higher barrier to inversion than imines.