Here is my calculation: note: $ν$ – frequency, $V$ – velocity of wave, $v'$– velocity of object. $$ \begin{align} λ &= h/mv' \tag{1}\\ mc^2/h &= ν \tag{2}\\ λν &= V \quad \text{(wave equation)} \tag{3} \end{align} $$
Substituting 1 and 2 in 3 we get:
$$h/mv' \cdot mc^2/h = V$$
Simplifying we get:
$$c^2/v' = V$$
However, according to the book, the answer is $hv/mc$. What is wrong with my answer?
I think the problem arises partially from the confusing notations. "Velocity of wave" $V$ is essentially $c$, "velocity of object" $v'$ is usually denoted with the speed of particle $v$ and the energy of photon $E$ is for some reason expressed via mass–energy equivalence $E = mc^2$ when it should be just kinetic energy of the particle $E = mv^2/2$ (but the latter is anyways irrelevant here).
So, your equation turns into this:
$$c^2/v' = V$$ $$c^2/v = c$$ $$c = v$$
which just states that the velocities of both wave and particle are equivalent. I suggest you just combine the de Broglie equation
$$λ = \frac{h}{mv}$$
and the relation between wavelength $λ$ and frequency $ν$
$$λ = \frac{c}{ν}$$
so that
$$λ = \frac{h}{mv} = \frac{c}{ν} \quad \implies \quad v =\frac{hν}{mc}$$
