The statement you read that "baking soda is a base" comes from the fact that a solution of sodium bicarbonate (baking soda) and water has a pH of around $8.3$.
However, sodium bicarbonate is amphoteric with respect to Brønsted–Lowry acid/base theory, which means that it can act as either an acid or a base. More specifically, the bicarbonate ion is amphiprotic, meaning that it can either donate a proton, acting as an acid, or accept a proton, making it act as a base. The reason that a sodium bicarbonate solution is slightly basic is that it's ability to take a proton from a water molecule is greater than its ability to donate a proton to a water molecule. In other words, the chemical reaction causing the solution to be basic is:
$$\ce{HCO3- + H2O <--> H2CO3 + OH-}$$
The other equilibrium reaction at play, the reaction "trying" to make the system more acidic, is:
$$\ce{HCO3- + H2O <--> CO3^2- + H3O+}$$
Again, in an otherwise pure aqueous system, it is the first reaction that dominates, and thus giving the observed slightly basic solution.
Note that the $\ce{H2CO3}$ produced in the first reaction is also part of yet another equilibrium process:
$$\ce{H2CO3 <--> CO2 ^ + H2O}$$
Where the $\ce{CO2}$ is liberated as a gas. This is the cause of the well-known "fizzing" when baking soda is added to vinegar (acetic acid).