I can only really speak for bromine(VII). Bromine(V) is pretty common and I'm not entirely sure what's the deal with Br(I) readily disproportionating to Br(V) + Br(-1). (The tendency of Br(I) to undergo disproportionation explains why we don't see it very commonly, but why it has this tendency I'm not so sure.)
This is an example of what is sometimes called the "alternation effect", or "d-block contraction" in p-block chemistry.
Due to the intervention of the ten 3d electrons, which are poorly shielding, the 4s and 4p electrons of the elements Ga through Kr are more tightly bound than one might expect. These elements show a distinct reluctance to take on their group valency. In $\ce{BrO4-}$, the higher-than-expected energy cost of ionisation to form Br(VII) is not fully compensated for by covalent bond formation (the strengths of which decrease linearly down the group). A selection of other observations that can, to various extents, be attributed to this are:
- the anomalously high electronegativity of germanium (2.01), compared to silicon (1.90) and tin (1.96)
- the anomalous instability of $\ce{AsCl5}$ (decomposes above $-50~\mathrm{^\circ C}$) even though $\ce{PCl5}$ and $\ce{SbCl5}$ are well-known,
- the fact that nitric acid oxidises sulfur to $\ce{H2SO4}$, but selenium only to $\ce{H2SeO3}$
In a sense, it is exactly the same as the inert pair effect, which arises partly due to the lanthanide contraction - i.e. the intervention of the fourteen 4f electrons, which shield 5d, 6s and 6p electrons poorly. The difference is that the inert pair effect is even more pronounced because of significant relativistic stabilisation of the 6s electrons. Here's an insightful discussion on the topic: Why does tin form tin (II) compounds?
