Yes, and no.
Acid and base chemistry is a neat balance between equilibria. As you well know, water dissociates readily by itself. A tendency that can be observed in any compound with $\ce{-OH}$ groups, yet not to a similar extend.
In other words, a hydroxyl group tends to be slightly acidic. Given a base, i.e. sodium hydroxide in this particular case, the following equilibrium will develop:
$$\ce{EtOH + NaOH <<=> EtO- + Na+ + H2O}$$
While the equilibrium will mostly be on the left side, depending on what other substrates are available, a tiny amount of alkoxide might already be enough to drive the reaction forward. In this case it is important to look at what actually drives the reaction.
In the example of 4-bromo-1-butanol that your book uses I am assuming there is a cyclisation happening.
$$\ce{BrH2C(CH2)2CH2OH + NaOH <=>> THF + H2O + NaBr}$$
Unfortunately I cannot quickly find a reference, but sodium bromide is not very soluble in diethylether, so I would assume it's not very soluble in THF either. The driving force here would be having a well enough leaving group and the continuous removal of product, basically making it a one-way reaction.
In some cases you need a higher concentration of alkoxide, and that's where hydrides come into play.
$$\ce{EtOH + NaH <=>> EtO^-\bond{~}Na+ + H2 ^}$$
In this case dihydrogen, as it is a gas, will be removed from the reaction mixture, basically ensuring full conversion to the alkoxide.