No, it doesn't contradict the ideal gas law at all. When you are saying that rise in volume means increase of temperature, you are inherently assuming that Pressure remains constant. But that is not the case at all in an adiabetic expansion.
In fact, you should realise that it is the product $PV$ which is decreasing. So, Volume might increase, but pressure drops even faster than volume, that's why the product $PV$ overall decreases. If some of the weights are removed, the gas expands slowly according to the equation, $PV^{\gamma} = C$. For most of the gases $\gamma > 1 .(\gamma = \frac{C_p}{C_V})$ Thus by simple observation, you can see if Volume increases $x$ times, Pressure will decrease $x^{\gamma}$ times, which is more dominating in the product.Thus, the product $PV$ will definitely decrease with increase in volume in case of an adiabetic process.
Also, your second argument is also not acceptable, because you already realise that pressure will decrease and by Kinetic theory of ideal gases $PV = \frac{1}{3}m\ n\ v^2_{rms}$. Thus if you understand the decrease in the product $PV$, you will also understand that $v^2_{rms}$ will also decrease, and hence the kinetic energy of gas molecules also.
Also you are thinking in a much more complicated way towards the situation. Think simply. For ideal gases, internal energy ($U$) is a measure of the translational kinetic energy of gas molecules and also $U = \frac{3}{2}RT$ depicting that for ideal gases, kinetic energy is only a function of temperature only. Also, by the first law of thermodynamics, $dU = \delta q + \delta W$ ( $\delta $ here represents inexact differentials). For adiabetic process, $\delta q =0$. So, $dU = \delta W$. Now, $\delta W$ is work done on the system which is negative for an adiabetic expansion, which makes $dU$ also negative, i.e. internal energy decreases and so does temperature.