Take the fundamental equation for the Gibbs free energy of a system:
$$dG=VdP-SdT+\sum_i\mu_idn_i$$
Thus, the chemical potential can be defined in terms of the Gibbs free energy as follows:
$$\mu_i=(\frac{\partial G}{\partial n_i})_{P,T}$$
Now, assuming that the chemical potential is positive (i'm not sure if that is always true) if I add $dn$ of component $i$, $dG>0$ which means (given that the Gibbs free energy is the capacity of a system to do non PV work) I have enabled the system to do more work. Consequently, I have done work on the system by adding more substance to it because i think I have increased it's potential - like raising a mass to a greater height.
It makes sense to me that adding a substance to a system would increase it's capacity to do work however, it doesn't make sense to me that work must be done to add moles of a substance. It's more clear when I think about the converse...
If I remove a small amount of the substance I reduce it's capacity for work and the system must therefore give out energy. This is confusing to me because surely i'd have to do work to remove a portion of a substance from a system - breaking the intramolecular forces and so on. However, for an ideal gas, there aren't any intramolecular interactions so unless the chemical potential for an ideal gas is zero (which I assume it's not) I must be missing something?