Assuming the Van der Waals equation with $b=0$, I agree with the solution to the equation:
$$pV_\mathrm{m}^2 - 24V_\mathrm{m} + 2 = 0\label{eqn:1}\tag{1}$$
I'll point out that I'm following the notation of the given solution, but $V_\mathrm{m}$ seems odd to me. I'd think that $V_\mathrm{m}$ would be reserved to mean the molar volume at STP.
But equation \eqref{eqn:1} has two unknowns, $p$ and $V_\mathrm{m}$. Letting $a' = p$, $b' = -24$, and $c' = 2$ then the appropriate quadratic equation is of course
\begin{alignat}
2V_\mathrm{m} &= \dfrac{-b' \pm\sqrt{b'^2 - 4a'c'}}{2a'}\tag{2}\\
V_\mathrm{m} &= \dfrac{-(-24) \pm\sqrt{(-24)^2 - 4×p×2}}{2p}\tag{3}\\
V_\mathrm{m} &= \dfrac{24 \pm\sqrt{(-24)^2 - 8p}}{2p}\tag{4}\\
V_\mathrm{m} &= \dfrac{12 \pm\sqrt{144 - 2p}}{p}\label{eqn:5}\tag{5}
\end{alignat}
So we have two unknowns and one equation. Thus the problem is unsolvable without additional information. Notice that would still be the case if the simple ideal gas equation, $PV=nRT$, had been used. The ideal gas equation could be solved only to $pV_\mathrm{m}=24$.
For equation \eqref{eqn:5} there are three cases for the sqrt term $\sqrt{144 - 2p}$:
The term is negative.
If $144 - 2p$ is negative, then both roots would be imaginary.
The term is equal to zero.
We have $\sqrt{144 - 2p} = 0\implies 144 - 2p = 0,$ so $p = 72$. Therefore, $V_\mathrm{m} = 12/72 = 0.167.$
The term is greater than zero.
Then $72 > p$ also, and mathematically $V_\mathrm{m}$ has two roots.
Now, again there are three possibilities.
(a) $\sqrt{144 - 2p} < 12$
This is impossible since $V_\mathrm{m}$ would have two valid values.
(b) $\sqrt{144 - 2p} = 12$
This is impossible since $V_\mathrm{m} = 0$ and the gas volume can't be $0$.
(c) $\sqrt{144 - 2p} > 12$
This would yield one negative value and one positive value for $V_\mathrm{m}$, which is OK for $V_\mathrm{m}$. However it also means that $p < 0,$ which is nonsensical.
Thus the only reasonable solution is if the square root term is equal to zero.
P.S. The comment by user Poutnik made me look at this again, and I now understand the book's solution…