As you correctly pointed out, you can connect the moles of each compound in the gas phase, $a$ and $b$, to the composition of the liquid phase, $A-a$ and $B-b$, through their partial pressures, $p_A$ and $p_B$:
$p_A=a\cfrac{RT}{V}=p_A^o\cfrac{A-a}{\left(A-a\right)+\left(B-b\right)}$
$p_B=b\cfrac{RT}{V}=p_B^o\cfrac{B-b}{\left(A-a\right)+\left(B-b\right)}$
And, as you did, we can use the total number of moles in the gas phase to rewrite these equations in terms of $p_e$. Since
$p_e=p_A+p_B=\left(a+b\right)\cfrac{RT}{V}$
and therefore
$\cfrac{RT}{V}=\cfrac{p_e}{a+b}$
we can write
$p_e\cfrac{a}{a+b}=p_A^o\cfrac{A-a}{\left(A-a\right)+\left(B-b\right)}$
$p_e\cfrac{b}{a+b}=p_B^o\cfrac{B-b}{\left(A-a\right)+\left(B-b\right)}$
Note that this is a two-equation system with two unknowns, so, barring incompatible values of the constants $A$, $B$, $p_e$, $p_A^o$ and $p_B^o$, it's determined (as you essentially pointed out).
Let's find a relationship between $a$ and $b$. A straightforward way to do that is by dividing the equations:
$\cfrac{p_e\cfrac{a}{a+b}}{p_e\cfrac{b}{a+b}}=\cfrac{p_A^o\cfrac{A-a}{\left(A-a\right)+\left(B-b\right)}}{p_B^o\cfrac{B-b}{\left(A-a\right)+\left(B-b\right)}}$
$\cfrac{a}{b}=\cfrac{p_A^o}{p_B^o}\cfrac{A-a}{B-b}$
This expression is key; we will return to it later.
Clearing $a$:
$a=\cfrac{p_A^o}{p_B^o}\cfrac{b}{B-b}\left(A-a\right)$
$a\left(1+\cfrac{p_A^o}{p_B^o}\cfrac{b}{B-b}\right)=\cfrac{p_A^o}{p_B^o}\cfrac{b}{B-b}A$
$a=\cfrac{\cfrac{p_A^o}{p_B^o}\cfrac{b}{B-b}}{1+\cfrac{p_A^o}{p_B^o}\cfrac{b}{B-b}}A=\cfrac{b}{\cfrac{p_B^o}{p_A^o}\left(B-b\right)+b}A$
We haven't made any additional assumptions in the derivation (beyond Raoult's law itself), so this is valid for all the situations where Raoult holds. Let's interpret the numbers to see what they mean.
With this expression you can check out various situations to get a feeling of how does Raoult behave. Some limit cases:
· If $p_A^o \approx p_B^o$, $\cfrac{a}{A} \approx \cfrac{b}{B}$ and both components are distributed to the same extent between the gas and liquid phase.
· If $p_A^o \gg p_B^o$, the vapour pressure quotient $p_B^o/p_A^o \ll 1$ and therefore $a \approx A$ (the more volatile component will be almost entirely in the gas phase).
· And, contrarily, if $p_A^o \ll p_B^o$, $a \approx 0$ (the less volatile component will be almost entirely in the liquid phase).
Let's return to the key expression above:
$\cfrac{a}{b}=\cfrac{p_A^o}{p_B^o}\cfrac{A-a}{B-b}$
If you ponder carefully what this means, it's telling you that for ideal mixtures (Raoult mixtures), the relationship between the components in the gas and liquid phases is always fixed as long as there is equilibrium, and the proportionality factor of that relationship is the vapour pressure ratio of the two components. So, if you have $p_A^o=2p_B^o$, the ratio of moles of A and B in the gas will always be twice the ratio of moles of A and B in the liquid. Or, equivalently, Raoult predicts that the gas phase is enriched in the most volatile component, precisely in proportion to the relative vapour pressures. The more volatile a component is, the more "additional fraction" of the gas phase it represents.
So, what values for $p_e$ are compatible with Raoult? Let's find out:
$p_e=p_A+p_B=p_A^o\cfrac{A-a}{\left(A-a\right)+\left(B-b\right)}+p_B^o\cfrac{B-b}{\left(A-a\right)+\left(B-b\right)}$
$p_e\left(\left(A-a\right)+\left(B-b\right)\right)=p_A^o\left(A-a\right)+p_B^o\left(B-b\right)$
$\left(A-a\right)\left(p_e-p_A^o\right)=\left(B-b\right)\left(p_B^o-p_e\right)$
$\cfrac{A-a}{B-b}=\cfrac{p_B^o-p_e}{p_e-p_A^o}$
Note that this is a particular case of the lever rule and, if all the quantities are positive, it demands that $p_A^o \leq p_e \leq p_B^o$ or $p_B^o \leq p_e \leq p_A^o$. A value of $p_e$ outside that range would correspond to a value of $a$ or $b$ outside of the range $0 \leq a \leq A$ or $0 \leq b \leq B$ - which doesn't make physical sense.
But is any value of $p_e$ between $p_A^o$ and $p_B^o$ valid? No, it depends on $A$ and $B$. Every value of $p_e$ corresponds to a certain composition - so you can't set both to arbitrary values.
Take for example $p_A^o=100$, $p_B^o=200$, $p_e=125$, $A=1$ and $B=4$. Applying the lever rule expression we just derived,
$\cfrac{A-a}{B-b}=\cfrac{p_B^o-p_e}{p_e-p_Ao}=\cfrac{200-125}{125-100}=3$
So, the ratio of moles of A to moles of B in the liquid in equilibrium is 3. In other words, 3/4 of the liquid molecules are of A.
Let's introduce this value in the key expression we derived earlier:
$\cfrac{a}{b}=\cfrac{p_A^o}{p_B^o}\cfrac{A-a}{B-b}=\cfrac{100}{200}3=1.5$
So, the ratio of moles of A to moles of B in the gas in equilibrium is 1.5. In other words, 3/5 of the gas molecules are of A.
All of this is compatible with Raoult's law... but it is incompatible with a total system composition of 1 mole of A and 4 moles of B!
Hope this helps.