In Carey, Sundberg Advanced Organic Chemistry it is written:
The relative slowness of the abstraction of protons from carbon acids by organolithium reagents is probably also due to the compact character of the carbon-lithium clusters. Since the electrons associated with the carbanion are tightly associated with the cluster of lithium cations, some activation energy is required to break the bond before the carbanion can act as a base. This kinetic sluggishness of organometallic compounds as bases permits important reactions in which the organometallic species acts as a nucleophile in preference to functioning as a strong base.
Won't the same reasons apply for nucleophilic displacement? Compactness of the aggregates should affect nucleophilicity, too. So why will BuLi add to a carbonyl instead of deprotonating the alpha carbon atom?