Your intuition is correct. The factor $A$ changes with temperature.
This article details how the value of $\ce{A}$ for an elementary, bimolecular reaction between $\ce{P}$ and $\ce{Q}$ can be derived to be:
$$A_{\ce{PQ}}=N_\ce{P}N_\ce{Q}d^2_{\ce{PQ}}\sqrt{\frac{8k_\mathrm{B}T}{\mu}}$$
The RHS is clearly a function of temperature. Without going into the details,[a] it suffices to remember that $A$ is a function of temperature because it is related to molecular collisions, which themselves are a function of temperature.
However, it is worth noting this paragraph on Wikipedia:
...under a wide range of practical conditions, the weak temperature dependence of the pre-exponential factor is negligible compared to the temperature dependence of the $\mathrm{e}^{(-E_\mathrm{a}/RT)}$ factor[b] (my emphasis); except in the case of "barrierless" diffusion-limited reactions, in which case the pre-exponential factor is dominant and is directly observable.
Given this, it may be within sufficient experimental errors to make the assumption that $A$ does not vary with temperature. However, it is just that, an assumption. In reality, $A$ does vary with temperature.
[a]: This isn't the exactly correct expression though. As the article itself notes, "Often times however, when the term is determined experimentally, $A$ is the preferred variable and when the constant is determined mathematically, $Z$ is the variable more often used. The derivation for $Z$, while mostly accurate, ignores the steric effect of molecules."
[b]: Of course, here Wikipedia is using the actually computed values of $A$ (and not vague estimated formulae like the one above; as actually $\sqrt{T}$ grows faster than $\mathrm{e}^{-1/T}$)