I have to assume things to answer this question. It would be great if you could include a picture of the actual stuff you have so that I may assess whether my assumption was correct or not.
The only aggregate state of desiccant silica gel, whether it be blue gel, orange gel or a colourless one, that I have seen to date is best described as a very coarse granulate. From these, unless you vividly shake the canisters repeatedly, there should be no dust evolution. Stuff like that is routinely stored and handled in chemical laboratories without anybody ever giving a damn because of dust — much unlike the very fine silica gel used for chromatography which can very much, as sixtytrees points out, cause silicosis.
The carcinogenic stuff inside blue gel is a cobalt compound. The salts of cobalt are only carcinogenic if you indigest them, and they do not permeate well through skin. They are even less prone to produce dust or carcinogenic volatiles even when heated to a hundred or so degrees. What they can do is stay on the skin after you touched the gel and then be transferred onwards, for example in your mouth on the crunchy apple you decided to eat — so wash your hands thoroughly if you accidentally handled the actual gel and did not use gloves.
Orange gel is slightly different, it uses organic moisture indicators. These have a much more significant vapour pressure than inorganic salts do. On the upside, these are also less carcinogenic. So the overall effect is probably similar.
Aside from that, you are at a very low risk. Sitting by a campfire is very likely more carcinogenic than putting blue gel desiccant into the oven for a few minutes.