Converting my comment into answer
Your query has been answered in this article. So I am going to write the relevant parts.
Nitrogen atoms are very small and have the ability to substitute for the carbon atoms in diamond's crystal structure. Trace amounts of nitrogen substituting for carbon in the diamond crystal lattice will cause the diamond crystal to selectively absorb blue light and selectively transmit yellow. This will cause those nitrogen-bearing diamonds to have a yellow color. Nitrogen is the most common impurity that substitutes for carbon and can comprise up to 1% of a diamond on the basis of mass.
Nitrogen can exist in the diamond crystal lattice in a number of ways. One way that influences color is when a single nitrogen atom is shared by four carbon tetrahedrons. This defect is known as the "C center" and is shown in the accompanying illustration. In this configuration, just one nitrogen atom per 100,000 carbon atoms can produce a noticeable yellow color in the crystal.
When nitrogen atoms substitute for carbon in the configuration described above, it produces a defect in the diamond crystal which alters how light passes through. The defect causes a selective absorption of blue light. The remainder of the spectrum is transmitted, and that results in the perception of yellow color in the eye of the observer.
Yellow color in mined diamonds can be also caused by the N3 defect. It consists of three nitrogen atoms clustered around a vacant carbon position in the diamond crystal lattice. When an N3 defect is accompanied by an N2 defect, certain wavelengths of blue and violet light are selectively absorbed by the diamond, and yellow light is selectively transmitted. This gives the diamond an apparent yellow color in the eye of the observer.
