It's a bit more involved than that; after all, the fluorine atom also has one of its electrons "moving down".
The key is to look at the occupied MOs and to determine whether they are mainly hydrogen-based, or fluorine-based (i.e., which atom contributes more to the MO). This is actually not trivial to determine at all, and in general, the qualitative MO diagram you've attached isn't going to be sufficient.
In this case, however, the bonding MO (labelled $\sigma$) is only constructed from two constituent AOs: one from hydrogen ($\mathrm{1s}$) and one from fluorine ($2\mathrm{p}_z$). The fluorine $2\mathrm{p}_x$ and $2\mathrm{p}_y$ AOs are nonbonding.
Under these circumstances, it's possible to look at the energy levels to indirectly determine the contributions. In this case, the bonding MO is closer in energy to the fluorine AO than it is to the hydrogen AO. This suggests that the contribution of the fluorine AO is larger; often we say that the bonding MO is "centred on fluorine" or "localised on fluorine".* Effectively, the electrons in this MO would belong more to fluorine than they do to hydrogen.
Note that this can't be generalised to more complex molecules and MO diagrams. It only truly works in the specific case where two AOs make up two MOs. Even, for example, carbon monoxide is problematic: it is only formed from two atoms, but its MOs generally have contributions from more than two different orbitals (carbon 2s and 2p, and oxygen 2s and 2p). So this simplification falls apart.
* This can all be formalised properly using quantum mechanics and linear algebra, but I won't do it here. The treatment would be similar to that which I showed here, except that you would need to explicitly calculate the actual eigenvectors. It should follow from this that the lower-energy new eigenvector (bonding MO) will have a larger coefficient of the lower-energy old eigenvector (AO).