Bond stability is the answer.
The energy required to break a $\ce{C-O}$ bond is $\pu{85.5 kcal/m}$; the energy given off when a $\ce{C=O}$ bond is formed is around $\pu{178 kcal/m}$ (reference). These bond energies suggest that a carbonyl will generally be a few $\pu{kcal/mol}$ more stable than the corresponding gem-diol $[(2 \cdot 85.5) - 178 = \pu{-7 kcal/mol}]$.
[Entropic effects also play a role (see @Martins comment), but while bond enthalpies will change within a series of compounds the entropic effects should be similar and roughly cancel out when we compare compounds.]
However, the $\pu{7 kcal/mol}$ difference noted above isn't large, so there is a delicate balance in the equilibrium between a carbonyl compound and its corresponding gem-diol. Since $\pu{1.4 kcal/mol}$ will shift an equilibrium by roughly a power of 10, small factors like dipole moments, inductive effects, etc., are enough to stabilize one side of the equilibrium more than the other and significantly shift the equilibrium from one side to the other.
If water is present, then an equilibrium will be established between a carbonyl compound and its corresponding gem-diol. Below is an interesting comparison of acetone and formaldehyde. A seemingly small change (changing $\ce{H}$ to $\ce{CH3}$) shifts the equilibrium from one side to the other (by 6 powers of 10!).
Here is a link to a Wikipedia article that has a few more interesting examples of carbonyl-diol equilibria.
In the case of chloral ($\ce{CCl3CHO}$) it is the strong electron withdrawing inductive effect of the $\ce{CCl3}$ group that destabilizes the already positively polarized carbonyl carbon driving the equilibrium to the diol side.
As an aside, if you're not familiar with the history of chloral (or as it is often called, chloral hydrate) google "Mickey Finn" or "knock out drops."
As I've mentioned above a change of just a few $\pu{kcal/mol}$ can markedly shift the equilibrium between a carbonyl and the corresponding gem-diol. Lest the reader think that gem-diols are a rarity, here are a number of examples of compounds that exist predominately in the gem-diol form:
- ninhydrin (because of the H-bond)
- $\ce{R-(CO)-(CO)-(CO)-R'}$ (middle ketone houses the gem-diol; see detailed explanation)
- 1,1-dihydroxycyclopropane ($\mathrm{sp^3}$ carbon relieves ring strain)
- floral hydrate (similar to chloral hydrate)