The other answers here, describing oxygen toxicity are telling what can go wrong if you have too much oxygen, but they are not describing two important concepts that should appear with their descriptions. Also, there is a basic safety issue with handling pressure tanks of high oxygen fraction.
An important property of breathed oxygen is its partial pressure. At normal conditions at sea level, the partial pressure of oxygen is about 0.21 atm. This is compatible with the widely known estimate that the atmosphere is about 78% nitrogen, 21% oxygen, and 1% "other". Partial pressures are added to give total pressure; this is Dalton's Law. As long as you don't use toxic gasses, you can replace the nitrogen and "other" with other gasses, like Helium, as long as you keep the partial pressure of oxygen near 0.21, and breathe the resulting mixtures without adverse effects.
There are two hazards that can be understood by considering the partial pressure of oxygen. If the partial pressure drops below about 0.16 atm, a normal person experiences hypoxia. This can happen by entering a room where oxygen has been removed. For instance, entering a room which has a constant source of nitrogen constantly displacing the room air, lowering the concentration -- and partial pressure -- of oxygen. Another way is to go to the tops of tall mountains. The total atmospheric pressure is lowered and the partial pressure of oxygen can be as low as 0.07 atm (summit of Mt. Everest) which is why very high altitude climbing requires carrying additional oxygen. Yet a third way is "horsing around" with Helium tanks -- repeatedly inhaling helium to produce very high pitched voices deprives the body of oxygen and the partial pressure of dissolved oxygen in the body falls, perhaps leading to loss of consciousness.
Alternatively, if the partial pressure rises above about 1.4 atm, a normal person experiences hyperoxia which can lead to oxygen toxicity (described in the other answers). At 1.6 atm the risk of central nervous system oxygen toxicity is very high. So, don't regulate the pressure that high? There's a problem. If you were to make a 10-foot long snorkel and dive to the bottom of a swimming pool to use it, you would fail to inhale. The pressure of air at your mouth would be about 1 atm, because the 10-foot column of air in the snorkel doesn't weigh very much. The pressure of water trying to squeeze the air out of you (like a tube of toothpaste) is about 1.3 atm. Your diaphragm is not strong enough to overcome the squeezing and fill your lungs with air. Divers overcome this problem by using a regulator (specifically, a demand valve), which allows the gas pressure at the outlet to be very near that of the ambient pressure. The principle job of the regulator is to reduce the very high pressure inside the tank to a much lower pressure at the outlet. The demand valve tries to only supply gas when the diver inhales and tries to supply it at very nearly ambient pressure. Notice that at depth the ambient pressure can be much greater than 1 atm, increasing by about 1 atm per 10 m (or 33 feet). If the regulator were to supply normal air at 2 atm pressure, the partial pressure of oxygen would be 0.42 atm. If at 3 atm, 0.63 atm. So as a diver descends, the partial pressure of oxygen automatically increases as a consequence of having to increase the gas pressure to allow the diver to inflate their lungs. Around 65 m (220 ft), the partial pressure of oxygen in an "air mix" would be high enough to risk hyperoxia and other dangerous consequences.
Now imagine a gas cylinder containing 100% oxygen. If we breathe from it at the surface, the partial pressure of oxygen is 1 atm -- high, but not dangerous. At a depth of 10 m, the partial pressure of supplied oxygen is 2 atm -- exceeding acceptable exposure limits. This is a general pattern -- raising the oxygen fraction of diving gasses decreases the maximum diving depth.
And you can't lower the partial pressure much because the lower limit, 0.16 atm, isn't that much lower than the 0.21 atm of sea level atmosphere.
One general category of solutions is to change gas mixes at various depths. This is complicated, requires a great deal of planning, and is outside the scope of your question. But it is certainly not as straightforward as just simplifying the gas mixtures or just raising the partial pressure of oxygen.
Additionally, compressed oxygen is a relatively annoying gas to work with. It is not itself flammable, but it makes every nearby organic thing flammable. For instance using grease or oil on or near an oxygen fitting risks spontaneously igniting the grease or oil. Merely having grease on your hand while handling oxygen refilling gear (with a small leak) can burn your hand.