I think the reason you're having trouble is your assumption that the operators are $S^2$ and $S_z$. In fact, those two operators do commute. The operators that do not commute are the individual components $S_x$, $S_y$ and $S_z$.
You also seem to have misunderstood the significance of the lack of "smear" result. When the silver ions make one pass and we do not see a smear, that simply tells us that angular momentum is quantized. If it could take on a continuous range of values, we would see the smear.
For the question of commutation, when two operators commute, it means that we can know the values of both at the same time. This necessitates that measuring one does not affect the value of the other. If the operators do not commute, we cannot know both their values simultaneously. Experimentally, this means that measuring one affects the value of the other.
The way the S-G experiment demonstrates that, for example, $S_x$ and $S_y$ do not commute is by first separating particles according to the value of angular momentum on one axis, for example $S_x$. So the output is one stream of particles with positive $S_x$ and one with negative. We then pass only one of these streams (say +x) through a magnetic field aligned to separate based on $S_y$. If the measurement of $S_y$ does not affect the value of $S_x$, then all of the particles should still have +x. However, passing the output of the $S_y$ "filter" into an $S_x$ filter again yields two spots, indicating that the $x$ component of the angular momentum has been scrambled by the measurement of the $y$ component.
Importantly, passing the +x output through an $S_x$ filter without first passing through an $S_y$ filter gives only the expected +x spot, indicating that the $x$ component values do not spontaneously scramble in the absence of the $S_y$ measurement.
Because the $S_y$ measurement scrambles the $S_x$ value, we cannot know both values simultaneously, and this is consistent with the mathematical result that the operators do not commute.
