You are right that normally a non-homogeneous system would be one where there is boundary between two internally homogeneous phases (like milk which is basically oil and water mixed at a very fine scale).
But Fermi as also allowing for the possibility that there may be only one phase where the concentration of its components varies continuously in space (if there is more than one phase the variation is discontinuous by definition).
It certainly isn't a common situation if the phase in in equilibrium where we would expect things like diffusion to create a uniform concentration of things over time, at least in liquids and gases). But it is very common in the real world where equilibrium is rarely reached (imagine the state of a poorly stirred cup of tea where the concentrations of tea-stuff will be larger near the tea bag than far away from the tea bag).
It is also very common in places where it is hard to get to equilibrium which is often true in solids (I don't know whether this is what Fermi was thinking of). Pyrex glass, for example, is apparently a single phase but the structure is deliberately made different near the surface than far away from the surface to create tension in the surface layers that protect the glass from thermal shock. But the properties vary continuously.
The examples I can think of may not be relevant to what Fermi was saying as they mostly are metastable and won't be at equilibrium which is often what thermodynamics restricts itself to worrying about.
