You are not right, Le Châtelier's principle plays a very important role.
Let's have a look at the reactions involving disodium monohydrogen phosphate, as it is an amphoteric substance and therefore a buffer on its own:
\begin{align}\ce{
Na2HPO4 + H2O &<=> 2Na+ + H3+O + PO4^3- \\
Na2HPO4 + H2O &<=> 2Na+ + {}^{-}OH + H2PO4^- \\
}\end{align}
To a small extent, there will also be the formation of phosphoric acid, but we'll ignore that for the moment.
Let's ignore the counter ion and formulate the equilibrium constants:
\begin{align}
\ce{HPO4^2- + H2O &<=> H3+O + PO4^3-} &
K_1 &=\frac{c(\ce{PO4^3-})\,c(\ce{H3+O})}{c(\ce{HPO4^2-})\, c(\ce{H2O})} &
= K_\mathrm{a}\cdot c(\ce{H2O})\\
\ce{HPO4^2- + H2O &<=> {}^{-}OH + H2PO4^-} &
K_2 &=\frac{c(\ce{H2PO4^-})\,c(\ce{{}^{-}OH})}{c(\ce{HPO4^2-})\, c(\ce{H2O})}&
= K_\mathrm{b}\cdot c(\ce{H2O})\\
\end{align}
The whole reaction can be described with a coupled equilibrium constant:
$$K = K_\mathrm{a}\cdot K_\mathrm{b}\cdot c^2(\ce{H2O})$$
At all times in a pure solution there will be phosphate, monohydrogen phosphate, dihydrogen phosphate and maybe even phosphoric acid in solution. Changing any of the concentrations of these species will change the equilibrium.
The same equations can be applied to the sodium dihydrogen phosphate system. The initial concentrations of course will be different.
Depending on the ratio of the two solutions, the pH will also change.
The buffer capacity is of course not only dependent on the overall concentration, but also if it is near the ideal buffer point.
Overall, polyprotic acids are very complicated systems and it solely depends on your understanding of the word "excellent" how the buffer behaves. There are many fine tuned buffer systems for various purposes, this website has quite a few neat tables.