If you actually label the stable isotopes of the elements involved ($\ce{^6Li,^7Li,^9Be,^{10}B,^{11}B}$) it becomes evident that $\ce{^8Be}$ is the missing stable isotope. What happens, as alluded to in the comments, is that pairs of alpha particles do not bond very well, much like pairs of noble gas atoms in ordinary chemistry except perhaps worse.
We often hear about "magic numbers" involving "filled nuclear shells" from physicists seeking stable superheavy elements, but these concepts extend down to lighter atoms and $\ce{^4He}$, an alpha particle, is actually the first of the highly stabilized, "doubly magic" nuclei. The stabilization of $\ce{^4He}$ by this property turns out especially effective in preventing pairing by nuclear forces unless you bring in either a three-way interaction (forming another relatively strongly stabilized $\ce{^{12}C}$ nucleus), or other particles such as an additional neutron.
So the two alpha-particle components of $\ce{^8Be}$ don't hold together; you need at least an additional neutron to hold the nucleus together against fission (a neutron works better than a proton because it avoids additional electrostatic repulsion). The stable isotopes of beryllium and boron (and heavier elements) have the additional particles they need to remain bound. Meanwhile for lithium the double-alpha splitting doesn't arise because you can't split off two alpha particles with only three protons.
