Most matches these days are safety matches: they're designed to need something more than ordinary levels of friction to ignite, by splitting the combustion materials between the match-head and the striking strip, i.e. the brownish paint thing down the side of the matchbox or across the front of the matchbook.
Most "regular" matches now are safety matches. The brownish strip has red phosphorous in it. The match head has potassium chlorate $\ce{KClO3}$. Combine those two with friction, and you get fire.
There are still striking matches, with heads of phosphorus sesquisulfide $\ce{P4S3}$, still around. Though largely in countries where consumer protection is not yet well-developed.
With sufficient energy, you can still ignite a safety match without the striking strip. But you'll need more energy than the friction of scratching it on a wall or bit of stubble.
(PS Don't play with fire. Get a friendly neighbourhood chemist to play with fire on your behalf.)