I guess the mistake you are doing is in the equation for $K_a$ for $\ce{CH3COOH}$. See if the following makes sense. 

$$\frac{[\ce{H+}]^2}{[\ce{CH3COOH}]}=K_a(\ce{CH3COOH})$$
Let $x$ be the $[\ce{H+}]$ from acetic acid, and $y$ from the other organic acid. 
$$\frac{10^{-2}}{0.1-x}=10^{-5}$$
$$0.09+x+y=0.1 \text{ (Balancing for H+)}$$
$$\frac{y^2}{[\ce{CHCl2COOH}]}=K_a(\ce{CHCl2COOH})$$

This should give you the answer.