In the $\ce{sp^2}$ transition state for an $\mathrm{S_{N}}2$ reaction
- the attacking nucleophile, the central carbon and the leaving group prefer a linear arrangement
- the 3 remaining groups around carbon are arranged with 120° bond angles
(image source)
Now, let's look at ethylene (as a model for a vinyl halide) and let's use the bent bond description of ethylene because it makes it easy to visualize what's happening (the bent bond model has been shown to be equivalent to the pi bond model in all aspects, see below for the same explanation phrased in terms of the pi bond model).
(image source)
Replace one of the ethylene hydrogens with halide and imagine what happens as we approach the transition state. The bond angle in the two-membered ring wants to be small, but as we approach the transition state it needs to open up wider to 120°! This would create a lot of strain. It is this large strain energy that causes the $\mathrm{S_{N}}2$ reaction at $\ce{sp^2}$ hybdridized vinylic carbons to be a very high energy pathway and therefore less likely to occur.
On the other hand, the elimination reaction at a vinylic carbon is quite facile. Here as the base removes a proton we build up negative charge on an $\ce{sp^2}$ hybridized carbon. An $\ce{sp^2}$ orbital has more s-character than an $\ce{sp^3}$ orbital; the more s-character in an orbital the lower the energy of an electron in that orbital. Therefore, negative charge is more stable in an $\ce{sp^2}$ orbital than a $\ce{sp^3}$ orbital. Consequently it is easier to remove a proton when the negative charge will reside on an $\ce{sp^2}$ hybridized carbon compared to an $\ce{sp^3}$ hybridized carbon (and that's why it is even easier [lower energy] to remove a proton from an $\ce{sp}$ hybridized carbon).
Explanation Based on Pi Bond Model
As expected for equivalent models, the bent bond explanation can be rephrased in terms of traditional $\ce{sp^2}$ sigma-pi hybridization, by noting that we start with an $\ce{sp^2}$ hybridized carbon which becomes $\ce{sp}$ hybridized in the transition state. It is known that $\ce{sp}$ hybridized carbons are roughly 5 kcal/m less stable than $\ce{sp^2}$ hybridized carbons (more electrons are in higher energy unhybridized p orbitals and this is not offset by the energy lowering experienced by the electrons as some bonds rehybridize from $\ce{sp^2}$ to $\ce{sp}$) thereby increasing the energy required to reach the transition state.