Nope, the rates wouldn't be affected, even if the nucleophiles used are different (though if you're dealing with highly concentrated reactants, it might be a different story...).
Judging from the way you framed the question, you probably already know a fair deal about both the Unimolecular and Bimolecular Nucleophilic Substitution (SN) reactions. So I won't start from scratch.
In the SN1 case, as the name would suggest, the rate depends on only one concentration factor, namely the substrate. You already (correctly) mentioned that a carbocation intermediate is formed. At this stage, its just a matter of whether the nucleophile performs a 'frontal attack' or a 'rear attack', and doesn't really involve the nucleophilicity or the size (at least not much in your example) of the nucleophile. This is different from the SN2 reaction, which requires species with high nucleophilicity and appropriate size to react (rear attack) with the substrate, leading to an unstable transition state.
How about an analogy?
Suppose you've got a lot of magnets stuck on your fridge door (at least I sure do..). Now suppose you're standing at a distance from the fridge, and you want to place another magnet on the fridge by throwing it. Since you've already got a lot of magnets stuck there, its going to be really hard to make your magnet land there. The weaker your magnet, the lower the chances of you landing it on the fridge door. However, if you have this really strong neodymium magnet, this wouldn't be much of a task; one throw, and it'll land so hard, it'll send the other magnets flying. This is a lot like the SN2 scenario. The attacking group must be a stronger nucleophile that the leaving group.
Now suppose your fridge door is devoid of magnets [which is precisely what my mother requires of our fridge.... :( ]. Then clearly throwing a magnet and hoping it'll land is a very, very good possibility. In this case it doesn't really matter whether the magnet is square or circular, large or small, strong or weak (not too weak). Apart from how large your fridge door is, there's no other factor that would keep the magnet from sticking to the fridge (albeit you aim). This is a lot like your SN1 scenario. The nucleophile's characteristics don't really matter (at least not in most typical cases). As long as there's space on the door, the magnet will stick.