The simple answer is that elemental, metallic mercury isn't very soluble in either the blood or in the contents of the digestive system. If ingested, metallic mercury will simple pass through the gut with little effect on the body and with very little being absorbed into the places where it can do harm. The same is true in the blood.
Mercury causes harm when it interacts with essential enzymes in the body and makes them malfunction. For it to do this, mercury atoms or ions have to be able to reach a significant concentration in bodily fluids. There are two ways for this to happen easily. Mercury vapour is absorbed into the bodily system (but this requires some effort such as combining bad ventilation with finely divided mercury droplets and heat or bad ventilation and a long time period as early mercury chemists discovered). Many mercuric compounds, especially organic ones, are also readily absorbed (either because they are soluble, as many salts are, or because they are volatile, as many organic mercury compounds like methyl mercury are.)
In fact the worst case of human mercury poisoning (the Minamata disaster) resulted from the release of methyl mercury into the environment where it accumulated in fish ultimately eaten by people.