One important concept you're missing here is that of dynamic equilibrium - That is, for equilibrium processes, reactions are going backwards at the same time as it's going forward.
So it's incorrect to think of a evaporation as a situation where water is going only from a puddle to the air. If there's any amount of water in the air (humidity), some of the water in the air goes "backwards" from the air into the puddle. The more water in the air, the more water in the air goes "back" into the puddle.
To a first order approximation, the forward reaction rate (evaporation) is related to the temperature and the surface area of the liquid. This isn't all there is to vapor pressure, though. The "pressure" in vapor pressure actually comes from the reverse reaction, where the rate is related to the temperature, the surface area of the liquid, and the number of water molecules per unit volume that are in the air. (The higher concentration of water in the air, the faster the reverse reaction rate.) Now, if we treat water vapor as an ideal gas, from the relation PV = nRT we can see that (at a given temperature) the number of water molecules per unit volume (n/V) is equivalent (within a scaling factor) to pressure: P = RT * n/V -- we call this the "partial pressure" of the substance.
What the vapor pressure is representing is the balance between these two effects (the forward and the reverse). If you fix the temperature and take the ratio of forward reaction rate and the reverse reaction rate (e.g. as you would to get a reaction quotient/equilibrium constant), you'll see that the effect of the surface area vanishes (as it's the same for both forward and reverse), and the only things left are temperature (which we're holding fixed) and the partial pressure of the water vapor. If we assume equilibrium, there's a particular value of the partial pressure which results in equilibrium at this temperature ... this is defined as the vapor pressure.
So the vapor pressure of a liquid is actually representing an equilibrium constant. At equilibrium, the amount of water vapor in the air will be equal to the vapor pressure. Have less water vapor in the air than the vapor pressure? Water will evaporate to bring the partial pressure of water in the air closer to the vapor pressure. Have more water vapor in the air than the vapor pressure? Water will condense out.
The other issue is that you're attempting to use an equilibrium concept (vapor pressure) in a non-equilibrium setting. While the general principles still apply - that with air less than the vapor pressure water will go from the liquid phase to the gaseous one - because you're continually replacing the air with new air you'll never actually reach equilibrium. In non-equilibrium settings you need to go beyond the equilibrium constant (the vapor pressure by itself) and look at the true reaction rates.
As we've already seen, the macroscopic reaction rate (the net rate of evaporation) is related to the balance between the two microscopic processes: the forward rate of evaporation and the reverse rate of condensation. While the rate of water molecules leaving the liquid into the air is related only to the temperature and surface area, the reverse reaction is also related to the amount of water in the air. This is why clothes dry slower in humid air at the same temperature: the (constant) forward rate of evaporation is offset by the humidity-dependent reverse rate. The more humid the air, the faster the reverse reaction rate, and the slower the clothes dry. (Though because you're below the vapor pressure, there's still a net positive reaction rate in the forward direction, and your clothes do dry eventually.)
P.S. You may be wondering why, if the reverse reaction happens even in air with less than 100% humidity, why don't we see water condense out on dry surfaces in those conditions? The answer is that it does. You will get small layers of water being deposited on surfaces, even in sub-saturating humidity. It's just that the forward reaction is faster, and they evaporate again before becoming noticeable. The other issue is that nucleation of liquid is a much slower processes than adding to an existing water layer, so it's much less noticeable than the slowing of the evaporation of existing water layers.