Thanks for asking this question. I'd heard before that Pu was actually more chemically toxic than its toxicity due to its radioactivity, but had never followed up by checking out this claim in detail.
tl;dr: Plutonium is very safe unless you grind it up into dust and inhale it, in which case the hazard is probably from radiation, not chemical toxicity. There is probably no way of knowing for sure, because we don't know enough about things like the dependence of harm from radiation on the dose rate as opposed to the total dose.
I'm going to assume we're talking about 239Pu, which is what is used in bombs. It has a half-life of 24,000 years, and it decays via alpha emission to 235U, which is effectively stable (half-life of almost a billion years). Note that it doesn't decay by fission (that only happens in bombs), so comparison with nuclear fallout is not relevant.
To get a comparison of chemical toxicity to toxicity due to radiation, I think all we need is a very rough order-of-magnitude estimate of chemical toxicity. Organic mercury is extremely toxic (about 10 times worse than arsenic), and it has an LD50 of about 0.1 g, so that's probably a reasonable upper limit for any other heavy metal such as plutonium.
239Pu's radioactivity is in the form of alpha particles, which are only dangerous if emitted internally, because they can't penetrate the epidermis. The hazard then seems to me like it would depend crucially on how the internal exposure occurred. If you eat an alpha emitter, then it will stay in your body until it's either excreted or decays. The half-life for excretion of organic or metallic Hg is on the order of months, so that's probably a reasonable order-of-magnitude estimate for other heavy metals. On the other hand, there are people at nuclear weapons labs who do machining of plutonium pieces, and this kicks up plutonium dust. They do this machining inside glove boxes, and I assume the chips and dust are swept up very carefully. The dust can be breathed in, and microgram particles that get into your lungs are likely to remain there for the rest of your life (Lenntech, ATSDR 2010). From the lungs, it can also migrate to the bones or liver. ("Much less than 1%" of ingested Pu would do so.)
For internal exposure to alpha emitters, we have good data on 210Po, which was used in the Litvinenko assassination. The LD50 for this substance is believed to be about 1 μg. It has a half-life of 138 days, and although I don't know its half-life for excretion, as described above I'd guess it's on the same order of magnitude, so to within a factor of order unity, we can probably pretend that all of it decays within the body. But if you eat 239Pu, then assuming it gets excreted within ~100 days, its probability of decaying within your body is only about $10^{-5}$. Therefore we would expect the LD50 for ingested 239Pu to be on the order of $10^5$ times greater than that for 210Po, or 0.1 g. This is on the same order of magnitude as the chemical toxicity. Well, this would be of interest if any human being (or lab animal) were ever going to be exposed to this much of the stuff, and then maybe we would be motivated to work out the biochemistry in more detail, do studies, and so on. But in reality, nobody has ever eaten or will ever eat this much plutonium, so it's of no interest.
For inhalation, the balance shifts. If someone lives for 50 years after exposure and carries the dust in their lungs for that whole time, then the probability of decay within the body is $\sim10^{-3}$. That means that in terms of radiation damage, the LD50 should be about 1000 times higher than for 210Po, i.e., about 1 mg. This would make it about 100 times more toxic in terms of radiation than any plausible chemical toxicity.
You could argue that plutonium might be less lethal than implied by this estimate because a radiation dose that would be lethal if suffered all at once can be less dangerous or almost harmless if spread out over a period of years. This probably reduces the radioactive toxicity of plutonium by a big factor relative to the estimate above, but I don't think it can reduce it by a factor of 100, which is what we would need in order to make the radioactive toxicity comparable to any realistic level of chemical toxicity.
So to summarize:
For external exposure: Zero hazard from radiation, some hazard from chemical toxicity but probably no worse than the (very small) risk from external exposure to substances like lead or mercury.
Ingestion: Chemical toxicity is not relevant because nobody will ever ingest enough to be in danger. On a population basis, radiation could conceivably cause some excess cancers, but I doubt that there is very much unreacted plutonium in fallout, and the population risk does not seem to have been considered sufficient to merit discussion in Simon et al.
Inhalation: The hazard from radiation is probably 1-100 times greater than the chemical toxicity is likely to be, even if the chemical toxicity is as bad as organic mercury.
References
ATSDR 2010, "Public Health Statement for Plutonium," https://www.atsdr.cdc.gov/PHS/PHS.asp?id=646&tid=119
Lenntech Water Treatment Solutions, "Health effects of plutonium," https://www.lenntech.com/periodic/elements/pu.htm
Simon et al., "Fallout from Nuclear Weapons Tests and Cancer Risks ," https://www.cancer.gov/about-cancer/causes-prevention/risk/radiation/fallout-pdf