Not all flame colours are determined by the metal, some anions also colour flames (see the linked question.)
Also, not all metals impart a notable flame colour (platinum, for example, is often used as a substrate material to do flame tests because it doesn't cause interference: small amounts of the substance of interest are deposited on a platinum wire which is then inserted into the flame).
What actually causes colour are the electron transitions where excited electrons fall back to lower energy levels emitting light in the process. Flames cause enough excitation that this process is common. Some metals have strong electronic transitions that match wavelengths of visible light. The bright yellow characteristic of sodium come mostly from electrons falling from the 3p to the 3s levels in sodium atoms and have a wavelength around 589nm. This is the same bright yellow that is visible in low pressure sodium street lighting (and is caused by exactly the same electronic process).
Other metals have different electronic structures and their emission spectra therefore have different colours. But not all metals have bright lines in the visible (the strongest lines in the spectrum of mercury, for example, are in the UV so won't give the same notable visible colour in a flame test). Many metals do, though, which is why flame tests are often useful. And even when you can't see the colour, instruments can, so spectroscopy is a useful analytical tool even when your eye can't see much.