I hear about things like dioxin, but I really don't see what's dangerous about plastics-shouldn't you be ble to just breathe it out, or pass it out of your system? How would it cause cancer?
The primary dangers from plastics themselves come when the monomer, and any compound put in to make the polymer matrix flexible, separate from the polymer and are released (leached) into whatever the plastic is containing. There are several such compounds which are known to be problematic to human biochemistry:
Phthalates - These are aromatic diesters, with a structure incorporating a phenyl ring with two ester linkages, typically to alkanes like ethyl or butyl groups. Phthalates are used in certain plastics to link the monomers, such as in poly-ethyl terephthalate (PET aka polyester), and in others as a plasticizer to make the normally rigid plastic flexible (PVC). Phthalates mimic certain hormones in the human body and so can cause issues with growth and maturation, as well as feminizing characteristics in males (reduced body hair, weight gain, moobs). PET is OK for packaging and storage of cold foods (it is, for instance, the plastic used for soda bottles), but you don't want to heat it because that will degrade the plastic to release the phthalate.
The ones used in PVC are especially problematic because they're not well-bonded; instead of being an integral piece of the polymer chain, they form weak hydrogen bonds between the polymer chains (thus the flexibility, even in relatively thick layers of the plastic), and so these phthalates easily leach out into water or "off-gas" into the air, even without heat. These phthalates are a big part of the "plastic smell" of new plastic products, and of water stored in them; the more flexible, and the longer the plastic has been kept enclosed, the more intense the smell. There was a big stink (NPI) in the early 90s when PVC was being used for water supply lines in home construction, and then they found all these problems with the stuff and a lot of homebuilders and chemical companies went bust paying for all this piping to be replaced with copper.
Bisphenols - These are variations of 2,2-bis-(dihydroxyphenyl)propane (which is the infamous Bisphenol A or BPA). Bisphenol A is the monomer for most of the original formulations of high-impact polycarbonate, such as Lexan. Again, when heated, the polymer breaks down and leaches the water-soluble monomer, which is structurally similar in many ways to phthalate and again has hormone-mimicking, endocrine-disrupting properties, which is why newer polycarbonates intended for food storage are made with other bisphenol monomers (most of which have similar endocrine properties, but because they aren't Bisphenol A they're still legal for now and allow the manufacturer to label the product "BPA-free").
Vinyl Chloride - The monomer of poly-vinyl chloride or PVC, this is typically a hazard in production facilities for PVC and when old, aged PVC begins to break down. The vinyl chloride monomer is a known carcinogen, which is the other reason PVC is no longer used for food packaging or water supply, and is being phased out from wastewater lines in favor of nylon.
As far as answering the question "shouldn't we just be able to pass it out of our system", the same could naively be asked of a whole host of chemicals that are bad for us. The biggest problem with that statement is the First Law of Toxicology; anything, in sufficient amounts, is harmful, even if it's absolutely essential for life in the right amounts. Selenium is an essential trace mineral in human biochemistry, required in certain protein synthesis mechanisms, but the LD50 blood concentration is about 3 mg/kg, and harmful effects such as stunted growth begin to be observed at concentrations within foodstuffs of as low as 4 to 5 ppm. Heck, you can die from drinking too much water, faster than you could from lack of it (it causes a condition called hyponatremia where the added water dilutes electrolytes to the point the body can no longer function).
What causes many chemicals to be harmful is in fact our body's own affinity for them as a beneficial chemical. Either the chemical is beneficial to us, or it looks like one that is, and the chemical is normally so rare in our diet and environment that our bodies grab all of it that they find, without an "off switch" to regulate intake.
Which leads to the second problem with that statement; sometimes our bodies don't efficiently metabolize and dispose of chemicals. Vitamin A, for instance, is an essential nutrient, and our body absorbs all it can, and gets rid of it only very slowly. As a result, it "bioaccumulates", and eventually becomes toxic. That's why beta carotene is the preferred "form" of Vitamin A in supplements; it's in fact an intermediate that our bodies can use to synthesize the vitamin as needed, but being water-soluble unlike actual Vitamin A, any unused excess of beta carotene is simply flushed out via the kidneys.
That process in itself can cause problems; the latest round of drug recalls the lawyers are running commercials on are for drugs that did exactly what they were supposed to, but were then metabolized into carcinogenic compounds that concentrated in the urine and caused various cancers of the excretory system.
And the last problem, tying back into the case in point, is that even if the body selectively absorbs it, efficiently processes it, and it isn't acutely toxic in any form or intermediate along the way, as long as there's an input of the chemical into your system, you'll always have some of it. Plastics are ubiquitous in consumer products, and especially food prep and packaging. As a result, virtually every resident of the industrialized world has a detectable amount of artificial phthalates in their blood, because it is simply impossible not to come into contact with them. Even USDA-certified organic foods have phthalates in them, because the cows eat food that's been stored or served in plastic bins and it shows up in their milk and meat, or the vegetable was fertilized with compost stored in a plastic bin or watered from a PVC irrigation system (even the farmer's market items from someone's backyard were likely grown using water collected in a plastic rain catchment and/or dispensed from a hose or a drip irrigator system that leaches plasticizer). What you store it in or prepare it with once it gets to you is immaterial; the damage is done.