I know that certain elements like plutonium and certain isotopes of caesium are radioactive but what gives them this ability? If it's their electron/proton/neutron arrangement, what makes that arrangement give radioactive properties and how?
Radioactivity is not at all dependent on the atomic or molecular structure: it a purely nuclear phenomenon. The only thing that is relevant is the structure of the nucleus.
The nucleus contains only protons and neutrons and the distinguishing feature that makes nuclei radioactive is, crudely, the ratio of the two. Some arrangements are unstable for a variety of reasons a full understanding of which requires fairly advanced quantum theory. What you need to remember is that some ratios or arrangements of protons and neutrons are inherently unstable and spontaneously decay in a variety of ways to more stable nuclei. For example, 39K (which contains 19 protons and 20 neutrons) is stable but adding a neutron makes 40K which is slightly unstable and spontaneously decays in several possible ways. You can tell this isn't a simple phenomenon, though, as 41K, with two extra neutrons is stable again.
It is also important to note that radioactive decay is essentially uninfluenced by anything outside the nucleus. For any given nucleus there is a probability that it will decay but that event occurs randomly and we can't tell when it will happen (but the rate of decay is stable and not influenced by temperature, pressure or the chemical compounds the nuclei are present in).