- What is the origin of the second formula (2)?
Equation (2) evidently refers to the free energy of formation of a solid solution of A and B, in which case neither element is in its standard state as a product (so that $a_i\neq 1$). You can rewrite expression (2) as follows:
$$\Delta G_\mathrm{f}^\circ = NRT\ln{\left(a_\ce{A}^{x_\ce{A}}a_\ce{B}^{x_\ce{B}}\right)} = RT\ln{\left(a_\ce{A}^{x_\ce{A}}a_\ce{B}^{x_\ce{B}}\right)^N}= RT\ln{\left(a_\ce{A}^{Nx_\ce{A}}a_\ce{B}^{Nx_\ce{B}}\right)}= RT\ln{\left(a_\ce{A}^{n_\ce{A}}a_\ce{B}^{n_\ce{B}}\right)}$$
$$ = n_\ce{A}RT\ln{a_\ce{A}+n_\ce{B}RT\ln{a_\ce{B}}}$$
Since the activity of the elements in pure solutions is 1 we can finally write
$$\Delta G_\mathrm{f}^\circ = n_\ce{A}\Delta G_{mA}+n_\ce{B}\Delta G_{mB}$$ where $\Delta G_{mi} = G_{mi}-G_{mi}^\circ= n_\ce{i}RT\ln{\left(\frac{a_\ce{i}}{a_\ce{i}^\circ}\right)}$.
- How does it come, that in one case the activity of the whole product AB is important and in another the single activities of the components of the product?
Equation (1) refers to a molar free energy of formation of $\ce{AB}$, whereas (2) refers to the free energy of formation of a solid solution of $n_A$ moles of A and $n_B$ moles of B. Equation (1) refers to combination of A and B at a 1:1 mole ratio, or equivalently reaction to form 1 mole of $\ce{AB}$ from $\mathrm{n_A=n_B=\frac12}\pu{mol}$. Reaction (2) refers to mixture of A and B at any arbitrary ratio or total number of moles. Therefore equation (2) is more general.