Colligative properties depend solely on the number of even though the interactive forces are different for different solute-solvent pairs. So why is the dependence only on the number of solute?
It's a lie. Colligative properties do depend on the chemical nature of the solute and solvent - their interaction. The trick is: an ideal solution, in which the solute-solute intermolecular forces and solute-solvent interactions are more or less of the same character, has the property that colligative properties are dictated by composition given only in terms of amount of moles. However, any sufficiently diluted solution will behave somehow ideally, that is where the statement goes: for diluted solutions, colligative properties depend only on the number of moles of solute.
A very simple, qualitative explanation:
After your solute has dissolved, there are no more enthalpic effects to take into account. The solvation enthalpy is converted to a temperature change, and that's it. For colligative properties, the solute ideally has no significant own vapour pressure, does not precipitate, it just stays in solution.
The solute molecule moves around (Brownian motion), the solvent molecules around it exchange places, but are all the same, and that's that.
Now everything in solution is about entropy. In a solution, there is only orientational and translational entropy. (Molecular) Vibrational excitation is only relevant at high temperatures, rotation is practically imposible due to the many interactions.
That's why every solute particle, irrespective of its size, has the same contribution to the entropy.
If the solute concentration inceases, its particles start interacting. Their coming close to each other and detaching again makes local energy fluctuations, which add entropy, but depend strongly on the kind of interaction. That's when the identity of the solute starts to matter.
Colligative properties are defined as solvent properties that display a linear dependence on the mole fraction $x_2$ of solute in solution, but are otherwise blind to the specific nature of intermolecular interactions. The key condition for such properties to be observed is that Raoult's law (one definition of an ideal solution) holds. Colligative behavior is purely statistical. For colligative properties - including freezing point depression, boiling point elevation, and osmotic pressure - the change in the property of the solvent can be expressed as $\Delta J=K_J x_2$, where $K_J$ is regarded as a constant for the solvent at the given temperature and pressure, independently of the identity of the solute. The relations assume explicitly that the solute concentration is low ($x_2\lt\lt1$). In practice, the requirement of dilute solutions is often necessary to ensure that Raoult's law is observed.
Raoult's law implies that the chemical potential of the solvent, $\mu_1(l)$, can be written as
$\mu_1 (l) = \mu_1^*(l)+RTlnx_1$
where $\mu_1^*$ is the chemical potential of the pure solvent at the same temperature and pressure. This equation is the basis for deriving the colligative property equations.
It should be emphasized that it is necessary to ensure ideal solution behavior - that Raoult's law is observed - in order to implement the mathematical formalism associated with colligative properties. In that sense this is a functional definition: one searches for systems that match the definition and allow application of the formalism.
When are these conditions met?
Quoting Atkins' physical chemistry textbook (Freeman, 4th Ed.):
some mixtures obey Raoult's law very well, especially when the components are chemically similar. [emphasis mine]
And while I'd rather you took my word for it, you can always check another tertiary source, the wikipedia:
In chemistry, colligative properties are properties of solutions that depend on the ratio of the number of solute particles to the number of solvent molecules in a solution, and not on the nature of the chemical species present.1 The number ratio can be related to the various units for concentration of solutions. The assumption that solution properties are independent of nature of solute particles is only exact for ideal solutions, and is approximate for dilute real solutions. In other words, colligative properties are a set of solution properties that can be reasonably approximated by assuming that the solution is ideal.