Yes, you are right, the reason primarily looking at it from a theoretical point of view is the high bond dissociation energy of $\ce{N#N}$
Let's look at this aspect from every point of view
Theoretical- High bond dissociation energy of $\ce{N#N}$
Kinetics- We have the Arrhenius equation given below, which states the relationship between the rate constant and activation energy of the reaction.
$$k = A \mathrm e^{-E_\mathrm a/(R T)}$$
So essentially, this relation shows that if activation energy is higher, then the rate is slower. However, if we increase the temperature sufficiently, then the reaction does start to happen significantly. (Hence, oxygen and nitrogen do react when lightning strikes)
- Thermodynamics- We have the very famous relation of thermodynamics
$$\Delta G= \Delta H-T \Delta S$$ So now lets again look at each term here
- $H$- Enthalpy of the reaction, it is very high, as we discussed before, the bond dissociation energy of $\ce{N#N}$ is very high.
- $S$- Entropy change won't be any significant as the reactants and products both are gases, so this is not an entropy-driven reaction.
Thus
, in every way possible we can conclude that nitrogen and oxygen do not normally react at room temperature.