In my textbook the molecularity is defined as(along with the standard definition) the order of the rate determining step(RDS). It is also given that the order can be a fraction but molecularity is always a whole number. But, then if the order of RDS is a fraction molecularity will be a fraction which is a against the definition. So, where is the problem? Is it that the order of RDS can only be a whole number?
What your book, and most other elementary books present, is at best a generalization, not applicable to all cases.
Molecularity, as per the collision theory, is defined only for elementary reactions. One of the things that define an elementary reaction is the fact that the entire transformation from reactants to products occurs simultaneously, and follows from a properly oriented, appropriately energetic collision of the reactant species. If the reaction is not a direct consequence of a single collision, it is not an elementary reaction, and hence molecularity is meaningless.
Now, what usually happens is that out of several elementary steps constituting a reaction, one is very slow as compared to others. So whatever the rate of that elementary reaction, becomes practically the rate of the entire reaction. And since it is an elementary reaction, its order is its molecularity, and to an approximation, the molecularity of the slowest step then becomes the rate of the equation.
But, in most cases, the slowest step is not slow enough for it to entirely determine the rate. It is the slowest step, and it is an elementary reaction and therefore its molecularity is its order. But now, its order is not necessarily the order of the entire reaction, since that was an approximation valid only in cases where the slowest step was so slow that it entirely determines the rate. Then, the order of the entire reaction, due to the influence of the other steps (elementary in itself), can be fractional, while the order of the slowest step, is still its integer molecularity. ( here the term slowest step is preferred over the misnomer rate determining step, as it does not entirely determine the rate).
Also keep in mind, that many a times what people refer to as the "rate determining step" or the "slowest step" is not an elementary reaction. then its order need not be an integer, even if it is slow enough for the order of the net reaction to be equal to the order of that step.