Electronic structure theory at the end of the day is "advanced quantum mechanics". IMO, one should understand the symbols from a QM perspective before applying it here.
Is $1/r_m$ an operator?
Recall that in QM, observables are represented by operators. Even the humble $x$ in classical mechanics is represented by an operator $\hat{x}$ in quantum mechanics. When acting on a wavefunction, the effect of the $\hat{x}$ operator is to multiply the wavefunction by $x$:
$$\hat{x}[\psi(x)] = x\cdot\psi(x)$$
Now, that's kind of boring, but it doesn't stop it from being an operator. Formally, an operator is just something which takes a wavefunction (aka a state, or a ket) and transforms it into another wavefunction. The action of $\hat{x}$ did exactly that: it transforms $\psi$ into $x\psi$.
Generally speaking, the same is true of $1/r$. The only difference is that your wavefunction now is expressed as a function of three spherical coordinates $(r, \theta, \phi)$. In the equation you wrote, there aren't any hats involved; this is mostly for notational convenience, and also because it barely makes any difference (notice how similar the LHS and RHS are on the above equation). You can imagine that there's a hat there, though, if you're more comfortable with that.
How do I start with a basis function and end up with a number that depends on the distance from the nucleus?
You integrate over all space, which reduces the function into a number. Presumably you know what $\mu$ and $\nu$ are, because this is your basis set which you have chosen.* These are both functions of $(r, \theta, \phi)$. Then:
$$\langle \mu | \frac{1}{r} | \nu\rangle = \int \left(\mu^* \cdot \frac{1}{r} \cdot \nu\right) \,\mathrm{d}V = \int_0^\pi \int_0^{2\pi} \int_0^\infty \left(\mu^* \cdot \frac{1}{r} \cdot \nu\right) r^2 \cos\theta \,\mathrm{d}r\,\mathrm{d}\theta\,\mathrm{d}\phi$$
Actually evaluating this integral with a computer is a totally different challenge, and not one that I'm particularly familiar with, but fundamentally speaking, the above integral tells you exactly what to do.
The distance of what, by the way?
The off-diagonal elements of QM operators ($\mu \neq \nu$) are quite hard to interpret classically, so I'd suggest to not think too much about it. At best one may view it as some kind of "interaction" term between the two states $\mu$ and $\nu$, but that generally does not yield much insight.
For the diagonal elements (i.e. $\mu = \nu$), and for a generic operator $\hat{A}$, it's possible to interpret the integral $\langle \mu | \hat{A} | \mu \rangle$ as representing the average value of the observable $A$ when the system is in the state $\mu$. (This is a fundamental postulate of QM, often one of the first things that is taught, so it's worth revising that if you don't remember it.) In this case the system is an electron, the state is $\mu$, and the observable is $1/r_m$ (distance to the $m$-th nucleus). So the term above may be interpreted as one over the average distance of an electron in the state $\mu$ from the $m$-th nucleus.
The basis function is not a point in space . . .
Indeed, and that's the whole point of performing the integration over all space (i.e., all parts of the functions $\mu$ and/or $\nu$ contribute to the integral).
* It's been a long time since I studied this; please correct me if this is wrong. It doesn't change the point, though. $\mu$ and $\nu$ should be functions that are fully known.
