What exactly is the difference between an ester linkage and a glycosidic linkage? I know both release a molecule of water, but what is the difference? Is it simply which types of monomers are involved in the bond?

In carbohydrates you can find examples of glycosidic linkages; the following example is α-maltose and the image is taken from Wikipedia, where a full list of authors is available.

This following example shows an ester; I chose one of the most simple esters (ethyl acetate) due to boredom. Again, the image is from Wikipedia, where a full list of authors is available.

One side of the molecule — the right-hand side in both of my examples here — always features a typical hydroxy group, i.e. is an alcohol. The difference is on the other side, the coupling partner. Look closely at maltose, and you may spot that the first carbon of the left-hand fragment also contains a hydrogen; its oxidation state is that of an aldehyde. Compare it with ethyl acetate, where the left-hand fragment linking carbon is in the oxidation state of a carboxylic acid (i.e. double bonded to an oxygen).

Technically speaking, glycosidic linkage is just a special case of acetal or ketal linkages; that is the correct name for the functional group $\ce{R^1O-CR^2R^3-OR^4}$ — acetal if either $\ce{R^2}$ or $\ce{R^3}$ is a hydrogen, otherwise ketal if both are carbons. ($\ce{R^1}$ and $\ce{R^4}$ must be carbon-based or we are dealing with hemiacetals/hemiketals/hydrates.) Examples of ketals for glycosidic linkages exist.

(And technically, ortho-esters exist, too, in which the linking carbon is bound to three $\ce{OR}$ groups. But these are very rare and special cases, even though they even exist in natural products such as tetrodotoxin.)

Simply and briefly put, esters’ vs glycosides’ “monomers”, or rather components, are (carboxylic) acids vs carbohydrates, and alcohols: