The short answer is that yes, the fact that many weak bases contain nitrogen does have to do with nitrogen itself. However, it's not really accurate to say that weak bases "usually" contain nitrogen.
For simplicity, I'm limiting my explanation to aqueous solutions, the Brønsted-Lowry definition, and main-group compounds. A base must do one of two things: either remove a proton ($\ce{H+}$) from water or dissociate to produce something else (typically $\ce{OH-}$) that can remove a proton from water. Most compounds that produce $\ce{OH-}$ directly are ionic; even if they don't dissolve much (like alkaline earth hydroxides), these ionic compounds dissociate completely and are therefore strong bases.
A base that removes protons from water must have a free pair of electrons for the proton to bond to. In period 2, boron tends to form compounds that tie up all three of its electrons. Carbon compounds, likewise, tend to have no free electron pairs. Oxygen and fluorine compounds (e.g., water and hydrogen fluoride) have free electron pairs, but because of their high electronegativity these elements keep their electrons to themselves and are unlikely to grab protons from water if they already have a full octet. Nitrogen is the only element in this period that balances two factors: it has a free electron pair in many of its compounds (e.g., ammonia and organic amines) and has a low enough electronegativity that it can share those electrons with a proton to form a cation (e.g., ammonium, $\text{NH}_3$).
Covalent compounds of elements in the third period and below tend to be very weak bases, if at all. I'm not sure why, but I suspect that it's because bonds involving these elements tend to form much weaker dipoles than bonds involving nitrogen. The weaker dipole would limit their ability to forcibly pull protons off of water molecules.
All that said, there are many ionic compounds that form basic solutions due to containing the cation of a weak acid. In these cases the base is a cation, not a molecule. Certainly some of these cations contain nitrogen, but many do not. Examples without nitrogen include any soluble carbonates and bicarbonates and any salts of carboxylic acids.