The bond dipole moment ($\mu$) of a chemical bond is equal to the magnitude of the separation of charge ($\delta$) times the displacement of the charge (or the distance over which the charges are spearated) $d$, or:
$$\vec{\mu}=\delta \vec{d}$$
Thus, there are two factors controlling dipole moment. The charge separation can be estimated from the electronegativity difference, but not perfectly. Disparities in size and polarizability also effect the separation of charge between two atoms. The displacement is the length of the bond. The $\ce{C-Cl}$ bond length (176 pm) is longer than the $\ce{C-F}$ bond length (134 pm) - data from here.
Thus:
For $\ce{C-F}$, the electronegativity difference on the Pauling scale is 1.43, and the bond length is 134 pm: $1.43\times 134 = 191$ (ignoring units at the moment).
For $\ce{C-Cl}$, the electronegativity different on the Pauling scale is 0.61, and the bond length is 174 pm: $0.61\times 174=106$.
So, my quick and dirty method did not work so well. However the disparity between these two bonds by my method $(106/191)=0.55)$ is smaller than by just comparing electronegativities $(0.61/1.43=0.42)$
and that is because of the other factor mentioned above: size and polarizability. Chlorine is bigger and more polarizable than fluorine (and carbon). The carbon-chlorine bond is polarized toward chlorine not just because of the electronegativity difference. Chlorine's larger size decreases the effectiveness of orbital overlap in bonding, and chlorine's increase polarizability implies that it can support more partial negative charge than suggested by its electronegativity alone.