Using molcalc.org to get a qualitative idea about what is going on, here are the charge distributions and the partial charge assigned to the electronegative N (-0.19) or O (-0.40):
Even with the quick cheap calculation done in the educational software, the calculated dipole moment of acetonitrile (4.05 D) is larger than that or acetaldehyde (3.16 D).
How do we rationalize a smaller dipole for acetaldehyde given the larger negative partial charge on the oxygen atom? You could consider the dipole of the methyl group separately from the dipole of the two functional groups. In acetonitrile, these dipoles line up, so they are additive. In acetaldehyde, they are at an angle, so the (vector) sum will be less.
If you look at all the calculated partial charges in acetonitrile, the hydrogen atoms are assigned most of the positive charge (3 x 0.08), and the nitrogen atom most of the negative charge. This results in a large charge separation, which influences the dipole moment (charge difference x distance). Surprisingly (to me), the carbonyl carbon is assigned a charge of -0.06.
If you look at all the calculated partial charges in acetaldehyde, the largest positive partial charge is assigned to the carbonyl carbon (+0.33), making for a shorter distance of the charge separation.
To get a more quantitative picture, one would have to perform a more serious quantum-chemical calculation. This would also be an opportunity to look at the orbital shapes and their relation to the charge distribution. This quick calculation, however, already points out some of the salient differences.
