Silver actually does prefer fluorine over heavier halogens. But this preference involves less (free) energy difference than the preference for the alkyl groups for fluorine. As a result, transfer of fluorine away from the silver has a net favorable energy output.
Let's look at some enthalpies of firmation (I cannot readily find free energy data for fluoromethane, so I had to use enthalpies).
$\ce{AgF}: -206\text{ kJ/mol}$ Source
$\ce{AgCl}: -127\text{ kJ/mol}$ Source
- Silver fluoride preferred by $79\text{ kJ/mol}$.
$\ce{CH3F}: -234\text{ kJ/mol}$ Source
$\ce{CH3Cl}: -102\text{ kJ/mol}$ Source
- Fluoromethane preferred by $132\text{ kJ/mol}$.
By this comparison the transfer of fluorine to the methyl group and chlorine to the silver is exothermic because of the stronger preference for the methyl group for the fluorine, the difference here being $53\text{ kJ/mol}$. The methyl group basically wins the alley fight for the fluorine because its muscles (energy for preferentially forming the fluoride) are stronger, leaving the silver to settle for second place (the chlorine).
A similar situation occurs naturally on Earth, with Group 16 elements. Practically all metals, even such metals as copper and lead which are usually found as sulfide ores, energetically prefer oxygen over sulfur. But their preference is less than what is seen with some other metals such as calcium, magnesium or even iron. So the latter metals are favored to form oxides, leaving sulfur predominantly for less thermodynamically selective metals like copper and lead.