One definite factor is the ability of the ligand to stabilize its lone pair. You observe that the cyanide ion is a better ligand than the nitrogen ligand. Let's think about this. The lone pair that reacts on the cyanide ion is the lone pair on the carbon atom. Not the lone pair on the nitrogen atom.
Carbon is less electronegative than nitrogen. Thus carbon is less able to stabilize its valence electron density, which in turns enhances the reactivity of its valence electron density through increased negative charge density.
Same with sulfide and oxide anions. Sulfur is less electronegative than oxygen, and thus less able to stabilize valence electron density. Hence the much greater stability of silver sulfide as opposed to silver oxide.
Other competing factors include charge density; the Lewis acid the ligand is coordinating to, medium (solvent), etc. For example the oxide anion is more basic than the sulfide anion in water solvent; i.e. the oxide anion forms a stronger bond with the hydrogen proton in water. But we just saw that silver sulfide is much more stable than silver oxide.