If a chiral molecule (an enantiomer) can react with itself, dimerize for example, to form a new compound, then formation of this new compound would contaminate the racemic mixture.

(image source)
Perhaps more importantly is interaction of the racemic mixture with itself in the solid state. Just as the interaction between a right hand and left hand is different than the interaction between two right hands, the interaction between the R form and S form of the enantiomer will be different than the interaction between two R (or two S) forms.
In liquids and gases, molecular encounters are fleeting and cannot be preserved or "locked in" for significant periods of time as they can be in a solid. In other words, any favorable, lower energy molecular encounters are quickly broken apart by the energetically larger effects of thermal motion. However, when the racemic mixture is in a solid phase, if the interaction between R and R (or S and S) is lower in energy then the interaction between R and S, then the R and S enantiomers will tend to form separate crystals and a mixture of pure R and pure S crystals will result. Such mixtures are often termed "conglomerates". On the other hand if the interaction between R and S is more favorable (lower in energy), then a racemic compound or racemate is formed where each crystal contains R-S pairs.
A conglomerate, being a simple mixture, will show a melting point depression compared to the pure enantiomer. A racemic compound may have a melting point higher or lower than the pure enantiomer.
Further reading on racemic conglomerates and racemic compounds can be found here and here.