Collision energy is what matters
Both the frequency of collisions and the average energy of those collisions are increased at higher temperatures. But the frequency doesn't matter much for the rate of reaction because collisions are already very very common and are not the limiting factor in causing a reaction to happen.
The distribution of energy in moving particles (and therefore the energy involved in collisions) follows a statistical distribution. Simplifying a bit (by ignoring the orientation of colliding molecules and a bunch of other probabilistic factors) the reason a reaction follows a collision is that the collision has enough energy to overcome some threshold energy barrier for a reaction to take place (simplifying again, collisions without enough energy just cause molecules to bounce off each other perhaps exchanging some kinetic or vibrational energy in the process). Only when the net energy involved is sufficient to, for example, break a bond in one of the molecules, does a reaction result. A lot of collisions are happening all the time and most don't have enough energy to cause a reaction to happen. In fact, if the population of collisions with enough energy is too low there will be no reaction at all no matter how many collisions there are and increasing the number will make no difference at all.
As the temperature increases, more of the molecules are pushed into the "have sufficient energy" part of the distribution (which might be a very small proportion of all the molecules in the mixture). Then reactions start to happen.
But the number of collisions is very large compared to the number leading to a reaction so it is the number of molecules with enough energy that dominates the result.
