If I went into specifics, this answer would span a few thousand books. Instead, I'll give an overview:
There are many ways of determining the products given the reagents:
Actually reacting the reactants
One can, of course, actually make the reaction take place. Once you've done that, you need to identify the contents of the resulting "soup". A few methods for this (You may have to separate the products first) :
Here, a nifty technique is used to determine the magnetic properties of a compound. From these properties, one can make reasonable guesses at the substructures existing in a molecule, and check against a list of possible products. This is a very powerful technique, and usually gives good results.
Here, we use various chemical reagents on the products, and analyse these products, making conclusions about the nature of the original products.
Here, using an STM, one figures out the structure of the molecule. This isn't used that often, but is an interesting technique.
There are many other such techniques, with varying accuracies and applicabilities.
Predicting the products in the absence of experiment
A lot of chemical research is into improving its (already vast) predictive power. The end-game of most sciences, after all, is to be able to predict stuff. So, in many cases, we can just predict how the reaction proceeds, without actually making the reaction occur in a lab.
This generally occurs in organic reactions. Here, you break the reaction into elementary steps. Each step can be explained by looking at electronic attraction/repulsion, as well as the increase in stability (which can generally be estimated by taking into account the inductive, mesomeric, hyperconjugation, and steric effects, though other things may effect the overall stability as well). From these steps, one gets a proposed mechanism, which may be verified.
Knowing solubilities and other numerical values
In ionic/inorganic reactions, the path a reaction takes generally hinges on the solubility of its final products. This is sometimes the case for other parameters as well. Knowing various numerical values lets one compare two possible reaction paths
Here, one uses the equations of quantum mechanics (Schrodinger's equation and a few of its cronies), and solves them for the given problem using numerical methods.
The particular reaction you have given
This one can be easily explained by resonance. In $\ce{CH3COOH}$, the $\ce{H}$ is loosely bonded, as the $\ce{C=O}$ bond resonates with the $\ce{C-O}$ bond. This makes it easy for the $\ce{H}$ to float away as $\ce{H+}$, leaving behind a pretty stable $\ce{CH3COO-}$ (made stable by resonance of the negatively charged lone pair over the oxygen atoms).
So far, we know that $$\ce{CH3COOH -> CH3COO- + H+}$$
Now we know that it is behaving like an acid. It is extremely well known that water forms a Hydronium ($\ce{H3O+}$) ion by self-dissociation, and this is accelerated by making the medium acidic (All free $\ce{H+}$s find the nearest $\ce{H2O}$ molecule and form $\ce{H3O+}$). $\ce{H3O+}$ is stable due to resonance, again.
So we now know that $$\ce{H+ + H2O -> H3O+}$$
Put the two reactions together and you get $$\ce{CH3COOH + H2O -> H3O+ + CH3COO-}$$
Though, in this case, the historical method of finding out the reaction may have been different -- acetic acid has been well known and studied for a very long time.
