Electronegativity is the power of an atom to attract bonding pairs of electrons to itself. It clearly depends on the nuclear charge: the larger it is, the more strongly the nucleus attracts electrons and the closer to the nucleus they stay, so the larger is the electronegativity. Besides, electronegativity also depends on the number of other electrons present in the atomic shells ahead of valence electrons participating in chemical bonding. These electrons shield the valence electrons from the positively charged nucleus decreasing its effective charge and consequently lowering the electronegativity of the atom.
Now, back to the question. $\mathrm{2s}$ orbitals are lower in energy than $\mathrm{2p}$ ones, so that electrons occupying the former are (at average) closer to the nucleus than that occupying the later. Thus, the more $\mathrm{s}$ character a hybrid orbital has, the lower its energy is and the closer (at average) electrons occupying it are to the nucleus. The lower the energy of a hybrid orbital of an $\ce{X}$ atom is, the lower is the energy of the molecular orbital in formation of which it participates, and consequently, the closer (at average) the bonding pair of electrons stay to that $\ce{X}$ atom.
So, with respect to a bonding pair of electrons that occupy a molecular orbital to which a hybrid orbital of an $\ce{X}$ atom contributes, the $\ce{X}$ atom indeed appears to be more electronegative as the $\mathrm{s}$ character increases. Specifically for $\ce{C}$ atoms, the larger the $\mathrm{s}$ character of a hybrid orbital is, the lower in energy is a bonding $\sigma_{\ce{C-C}}$ orbital formed from the hybrid ones: $E_{\sigma_\mathrm{sp}} < E_{\sigma_\mathrm{sp^2}} < E_{\sigma_\mathrm{sp^3}}$. So that electrons occupying the $\sigma_{\ce{C-C}}$ orbital stay closer to $\ce{C}$ atoms (at average) as the $\mathrm{s}$ character of hybrid orbitals increases.
With respect to the question OP asked in comments,
But if we compare between $\ce{C}$ and $\ce{H}$, is there some other factor to be accounted for because clearly H atom due to its very lower potential energy electrons should attract the shared electrons more.
the factor OP is looking for is the nuclear charge, which is 6(!) times larger for $\ce{C}$ atom comparing to the $\ce{H}$ one.