You can imagine following situation in this hydrolysis:
$$\ce{R-C(=O)NH2 + OH- <=>[$k_1$][$k_2$] R-C(NH2)(OH)O- \\->[$k_3$] R-C(=O)OH + H2N- ->[$k_4$] R-C(=O)O- + NH3}$$
Keep in mind that:
$$\ce{R-C(=O)NH2 + OH- ->[$k_1$] R-C(NH2)(OH)O- }$$
$$\ce{R-C(NH2)(OH)O- ->[$k_2$] R-C(=O)NH2 + OH-}$$
- In this reaction, $k_1 \gt k_2$ because of LeChateliers principle since $[\ce{OH-}]$ is in excess thus forward reaction is in favor.
- Also, $k_2 \gt k_3$ because of the reason given by OP ($\ce{OH-}$ is better leaving group than $\ce{H2N-}$).
- Since acid-base reaction is much faster than any other reaction, we can conclude that $k_4 \gt \gt k_3$.
Therefore, as soon as $\ce{R-C(=O)OH + H2N- }$ formed they would convert to $\ce{R-C(=O)O- + NH3}$ quickly so that $\ce{R-C(=O)OH + H2N- }$ can not able to go back to $\ce{ R-C(NH2)(OH)O-}$ intermediate. That means, when boiling with extra basic solutions, amide will be hydrolyze slowly to its corresponding carboxylate.