The description of $Z_\mathrm{eff}$ you gave is a bit too simplistic. As you correctly said, that would predict the same value of $Z_\mathrm{eff}$ for all elements in a group, which is not true.
In general we have $Z_\mathrm{eff} = Z - \sigma$ where $Z$ is the nuclear charge (that solely depends on the number of protons) and $\sigma$ is the shielding constant, which reflects the electron-electron repulsions but is not a simple function of the number of core electrons.
One of the earliest models to determine $\sigma$ was described by Slater in 1930.1 These are easily found online as "Slater's rules". These are quite simplistic, and so people tried to find better ways to calculate $\sigma$. Nowadays, one very popular source for values of $Z_\mathrm{eff}$ are the Clementi values.2 That's still quite long ago, so you can imagine that since then, we have come up with even more complicated ways to calculate it.
A peculiarity is that the values of $Z_\mathrm{eff}$ actually increase down the group. At the very least, that should dispel the myth that $Z_\mathrm{eff}$ only depends on the number of core electrons.
(Source: Keeler & Wothers, Chemical Structure & Reactivity: An Integrated Approach, p 264)
"Surely that should mean that the atomic radii decrease down the group!" you say.
I don't fault you for thinking that at all. However, that's not what we see: the atomic radii clearly increase going down the group. The second factor is the principal quantum number, $n$.
In the hydrogen atom, an electron in the 1s orbital has a much smaller radius3 than an electron in the 2s orbital. That's something that comes right out of the quantum mechanical description of the hydrogen atom.4 The same can be said for other atoms: a 2s orbital in lithium is smaller than a 3s orbital in lithium.
Note that I didn't compare a 2s orbital in lithium with a 3s orbital in sodium. Why? That's because sodium has a larger effective nuclear charge, and this serves to contract all the orbitals in sodium relative to those in lithium. Essentially, the orbitals are pulled inwards by the larger $Z_\mathrm{eff}$.
So, as you can see, going from lithium to sodium, there are two competing factors:
- Larger $n$ suggests that the outermost electron should be further away from the nucleus.
- Larger $Z_\mathrm{eff}$ suggests that the outermost electron should be nearer to the nucleus.
Unsurprisingly, it turns out that the variation of atomic radius is dependent on a combination of the two. We know that the atomic radius increases going down the group, so that must mean that the increase in $n$ (going from 2 to 3) outweighs the increase in $Z_\mathrm{eff}$ (going from 1.28 to 2.51), although it's sadly difficult to find an exact mathematical formulation in any textbook. It's likely that the exact dependency is very complicated and nowadays, such atomic properties are mostly calculated computationally anyway, which removes the need for such a mathematical formula.
Notes and references
1 Phys. Rev. 1930, 36 (1), 57
2 J. Chem. Phys. 1963, 38, 2686
3 I'm talking about $\langle r \rangle$, not the radius of a Bohr orbit, although that does also increase going from 1s to 2s.
4 I'm avoiding the circular logic that "higher energy means further away". It is further away because that is just the form of the radial wavefunction obtained by solving the Schrodinger equation - period.