The states $|LSJ\rangle$ are eigenstates of the $L^2$, $S^2$, and $J^2$ operators (note: these letters are operators) with the corresponding eigenvalues $L^2$, $S^2$, $J^2$ (these letters are quantum numbers used to label the states).
The long way around is to break the operators down into their Cartesian components. We have that $L^2 = L_x^2 + L_y^2 + L_z^2$ (this is a 3D Pythagoras theorem), and likewise $S^2 = S_x^2 + S_y^2 + S_z^2$, and $J^2 = J_x^2 + J_y^2 + J_z^2$. But also, because $\vec{J} = \vec{L} + \vec{S}$, we have that:
$$\begin{align}
J_x = L_x + S_x; \quad J_y = L_y + S_y; \quad J_z = L_z + S_z.
\end{align}$$
So:
$$\begin{align}
H_\text{so} &= \xi (\vec{L} \cdot \vec{S}) \\
&= \xi (L_xS_x + L_yS_y + L_zS_z) \\
&= \frac{\xi}{2} (2L_xS_x + 2L_yS_y + 2L_zS_z) \\
&= \frac{\xi}{2} [(L_x + S_x)^2 - L_x^2 - S_x^2 \\
&\quad\quad+ (L_y + S_y)^2 - L_y^2 - S_y^2 \\
&\quad\quad+ (L_z + S_z)^2 - L_z^2 - S_z^2] \\
&= \frac{\xi}{2} [J_x^2 + J_y^2 + J_z^2 - (L_x^2 + L_y^2 + L_z^2) - (S_x^2 + S_y^2 + S_z^2)] \\
&= \frac{\xi}{2} (J^2 - L^2 - S^2)
\end{align}$$
and so the corresponding expectation value is
$$\langle LSJ | H_\text{so} | LSJ \rangle = \left\langle LSJ \middle| \frac{\xi}{2} (J^2 - L^2 - S^2) \middle| LSJ \right\rangle = \frac{\xi}{2} (J^2 - L^2 - S^2).$$
The 'short' way, which is basically saying the same thing as above but in a much more concise manner, is:
$$\begin{align}
J^2 &= |\vec{L} + \vec{S}|^2 \\
&= L^2 + S^2 + 2(\vec{L}\cdot\vec{S}) \\
\vec{L}\cdot\vec{S} &= \frac{1}{2}(J^2 - L^2 - S^2) \\
\xi(\vec{L}\cdot\vec{S}) &= \frac{\xi}{2}(J^2 - L^2 - S^2) \\
\end{align}$$
but to convince yourself of the equality going from the the first and second line you need to expand the vectors out into their Cartesian components.
Addendum: Ian Bush has pointed out in the comments that this also assumes that the components of $\vec{L}$ and $\vec{S}$ commute with one another: otherwise, in principle we have that
$$(L_x + S_x)^2 = L_x^2 + S_x^2 + L_xS_x + S_xL_x$$
or
$$|\vec{L} + \vec{S}|^2 = L^2 + S^2 + (\vec{L} \cdot \vec{S}) + (\vec{S} \cdot \vec{L})$$
for the 'short' derivation. Thankfully, because $\vec{L}$ and $\vec{S}$ are different sources of angular momentum, they do fully commute, and so the last two terms in both lines above are equal. Thanks to Ian for the reminder!