If the Zn2+ ions aren't involved in the redox reaction, then why does the extra concentration affect the cell potential?
The overall reaction is
$$\ce{Zn(s) + Cu^2+(aq) -> Zn^2+(aq) + Cu(s)}$$
If this reaction were happening in a common reaction mixture (maybe with Zn and Cu powder on the bottom of a solution) we would write it as
$$\ce{Zn(s) + Cu^2+(aq) <=> Zn^2+(aq) + Cu(s)},$$
and would realize that the rate of the reverse reaction depends on the zinc ion concentration, thus affecting the equilibrium. However, the reaction at hand relies on electrons going through a single wire, and they would not go in both directions at the same time.
MaxW's comment is the key to the conundrum. There are reversible half reactions in either cell, and they determine the cell potential.
$$\ce{Zn^2+ + 2e- <=> Zn}$$
$$\ce{Cu <=> Cu^2+ + 2e-}$$
At each electrode, the half reactions are reversible, delivering electrons to the electrode and picking them up again. This back and forth establishes the potential difference that we can measure between the electrodes.
I wanted a conceptual explanation
For some, a kinetics-based answer (like the one above) counts as conceptual (or particular as opposed to macroscopic?). Here is a second way of thinking about it, using a different but related system. Let's consider a concentration cell with zinc electrodes and zinc ions in both cells. With higher concentration of zinc ions in the right cell than the left cell, the reaction would be:
$$\ce{Zn(left) + Zn^2+(right) -> Zn^2+(left) + Zn(right)}$$
Using your logic, the potential would not depend on the zinc ion concentration in the left cell because that is one of the products. Now we add some zinc ions (in the form of zinc chloride or such) to the left cell, making its concentration higher than that of the right cell. The reaction now would be:
$$\ce{Zn(left) + Zn^2+(right) <- Zn^2+(left) + Zn(right)}$$
Again using your logic, now the potential would not depend on the zinc ion concentration in the right cell because that is now the product.
The question is how the half reaction on the right would "know" to reverse direction, and to stop influencing the potential via its zinc ion concentration. I'm hoping this reductio ad absurdum (I had to look it up) is a convincing argument that zinc ion concentrations in both cells are relevant.