Hardness and toughness are not the same
Hardness and toughness are very different qualities in materials and are weakly related.
Hardness is strongly related to the more well-defined quantity of stiffness which measures how easily a compound can be deformed under stress. Glass and diamond are very stiff materials, for example. If you try to poke them with something they resist deforming to accommodate your poke. (Hardness is not perfectly aligned to stiffness because of small scale microstructures in many materials, but this is good enough for now).
Toughness is a vaguer term for materials and there isn't a simple way to measure it. This is partially because it varies depending on circumstances in a way that stiffness does not and it is a property of the overall structure and not just the materials that make up the structure. Understanding it requires some insight into why things break (and why other things don't). we need to know a lot about the small scale structure of materials not just the substances involved. For example, stainless steel (used in knives and forks) is a well-known tough material but cast iron is brittle. Both are mostly made of iron. The differences are in the crystalline structures.
One key property of tough structures is that cracks don't propagate. So a stick of glass will break easily as will a glass fibre. But a bundle of glass fibres embedded in an epoxy resin can be very tough (because the cracks in individual glass fibres are not propagated through the epoxy resin). Some tough metals can adjust the micro defects in their crystalline structures to absorb the strain that would otherwise propagate cracks. Some very soft compounds are very tough because the deform so easily that it is hard to start cracks, nylon rope for example.
Diamond is very very stiff. But it has no protection mechanism against cracks. So, like glass, once a crack has started it doesn't take a lot of energy to cause it to spread, so it may be stiff but it isn't tough.
A fuller explanation of this took material scientists and engineers a long time to work out would require a whole book to do it justice. Luckily that book has been written and is called The New Science of Strong Materials. It is well worth a read.