As noted earlier, the determining factor is the greater strength of the carbon-chlorine bond versus carbon-bromine, which leads to a higher activation energy for the magnesium to react with the former.
The strength of a carbon-halogen bond, and therefore the activation energy required for magnesium to form a Grignard reagent, is greatest with fluorine and decreases progressively with heavier halogens. The reactivity sequence between carbon-halogen bonds and magnesium may therefore be extended:
$\ce{C-F < C-Cl < C-Br < C-I}$
We might amend the first Inequality to read $<<$ rather than just $<$, as carbon-fluorine bonds are completely unreactive with normal procedures. Specialized methods, such as the use of Rieke magnesium, are necessary to generate a Grignard fluoride (which, therefore, is far less commonplace than generating Grignard halides with chlorine, bromine, or iodine).
The hydrocarbon group, too, can impact reactivity. Chlorine, bromine and iodine can all couple their nonbonding pairs with an aromatic ring, strengthening the bond of the halogen to the ring. Thus aryl halides react less rapidly than alkyl halides with a given halogen. Encyclopedia Britannica, combining the substrate and halogen effects reports:
Organohalogens vary greatly in their rates of reaction with magnesium. For example, alkyl iodides generally react very rapidly, whereas most aryl chlorides react very slowly, if at all.
This may be used to advantage, for instance if an aryl compound with both chlorine and bromine is exposed to magnesium, the chloroarylmagnesium bromide is formed with good selectivity.