First off, a "fishy" smell and rancidity are two different things. A fishy smell is likely due to impurities, primarily small volatile amines such as trimethylamine. These impurities are unrelated to oxidative degradation of the fish oil.
Rancidity in fish oil is the result of oxidation. Oxidation leads to the formation of lipid hydroperoxides and short-chain aldehydes, which can have "off" tastes and give rise to unpleasant odors, but in a very different way than the classic fishy smell.
Fish oil's primary components are docosahexaenoic acid and eicosapentaenoic acid. They are more sensitive to oxidation than many other fats because they contain many double bonds.
The best bet is to look for oil that is freshly prepared and highly purified. Some fish oils are esterified with ethanol to form ethyl esters right "on the boat". These ethyl esters can be distilled to remove most impurities. Optionally, the purified esters can be converted back to triacylglycerols to give "molecularly distilled" fish oil in the triglyceride form.