They caught fire because partly because they were milled in toluene and partly because they were such fine powders. As you probably know aluminum forms a passivating oxide on the surface. When you mill the powder, you increase the surface area which will oxidize when exposed to air. You milled it in toluene so oxidation did not occur during and after you were milling as long as it was covered in toluene.
$$\require{cancel}\begin{align}\\
\ce{Al (s) + O2 (g) &-> Al2O3 (s)} \qquad \qquad{\mathrm{\Delta H} = \pu{-1675.7 kJ mol-1}}\\
\ce{Al (s, toluene) + O2 (g) &\cancel{\ce{->}} Al2O3 (s)}
\end{align}$$
Once you filtered off the toluene parts and let it sit the toluene started evaporating and the aluminum started oxidizing. Once you turned the fan on you allowed the reaction to happen much faster to the point that it could not cool as it oxidized and the powder ignited. This is similar to smoldering coals, they are not glowing, but hot and still combusting. When you blow air on them the rush of oxygen will make them hot enough to glow again. This is essentially what turning the fan on did to your powder.