I understand that oxidation numbers are a method for keeping track of electrons in a reaction and how they are generally assigned (electrons in a bond are assigned to the more electronegative atom).
Why does this method work? Why does assigning these arbitrary numbers based on electronegativity work in predicting and balancing reactions?
First of all they are not really arbitrary.
But the main point, regarding balancing the equations is that: You can keep track of the total number of electrons per species, which is a well defined value. Like, for one permanganate ($\ce{MnO4-}$) you have 25+4*8+1 electrons, for one $\ce{Mn(II)}$ you have 23 electrons, for one water you have 10 electrons, for one $\ce{H+}$ you have 0 electrons. So for this half-equation, you must have 5 electrons on the left side:
$$\ce{8H+ + MnO4- + 5e- -> Mn(II) + 4H2O}$$
or you can say that, here I assume all H atoms are +, all oxygens are -2, and for consistency, Mn in $\ce{MnO4-}$ is +7. I can dismiss H and O's since all are same, and all change is because of Mn, which goes from +7 to +2 and difference is 5 electrons.
You could do this like that as well: O is -4 everywhere, so H's in $\ce{H2O}$ should be +2 and they were +1 on the left side. They donated 8 electrons. Mn in $\ce{MnO4-}$ should be +15 and it is +2 on the right side, which means it accepted 13 electrons. And the difference is 5 electrons, again.
Bottomline: It is all about consistency. If all elements in each and every species sum up to the charge of this species, whatever you say their formal charges are, it will work -because this just another way of keeping track of total electron count. So it is better to treat most of them constant like, O is -2, H is +1 almost always.
There are several ways to keep track of things. Oxidation number is one, formal charge is another.
With oxidation numbers, the electrons in a molecule or ion are assigned to the more electronegative atom in a bond. Thus in $\ce{H2O}$ oxygen is the most electronegative so the oxidation number of hydrogen is $+1$ (it has "lost" an electron) and oxygen is $-2$ (it has "gained" two electrons. Using this scheme in a consistent manner one can balance equations quite nicely.
On the other hand, there are times when the formal charge is more important. In this scheme one always splits the electrons in any bond between the two atoms. Thus in water all the atoms have a formal charge of zero.
One can think of these two schemes as follows: oxidation numbers pretend that all bonds are in effect totally ionic with the electrons going to the more electronegative atom. Formal charge pretends that all bonds are covalent and the electrons in a bond are shared equally.
Of course neither one is totally correct, but either one can be used as an "electron accounting" scheme.
