>1. What is the origin of the second formula (2)?

Equation (2) evidently refers to the free energy of formation of a solid solution of A and B, in which case neither element is in its standard state as a product (so that $a_i\neq 1$). You can rewrite expression (2) as follows:

$$\Delta G_\mathrm{f}^\circ = nRT\ln{\left(a_\ce{A}^{x_\ce{A}}a_\ce{B}^{x_\ce{B}}\right)} = RT\ln{\left(a_\ce{A}^{x_\ce{A}}a_\ce{B}^{x_\ce{B}}\right)^n}= RT\ln{\left(a_\ce{A}^{nx_\ce{A}}a_\ce{B}^{nx_\ce{B}}\right)}= RT\ln{\left(a_\ce{A}^{n_\ce{A}}a_\ce{B}^{n_\ce{B}}\right)}$$

$$ = n_\ce{A}RT\ln{a_\ce{A}+n_\ce{B}RT\ln{a_\ce{B}}}$$

where $n$ is the total number of moles ($n=n_A+n_B$).

Since the activity of the elements in pure solutions is 1 we can finally write

$$\Delta G_\mathrm{f}^\circ = n_\ce{A}\Delta G_{mA}+n_\ce{B}\Delta G_{mB}$$
where $\Delta G_{mi} = G_{mi}-G_{mi}^\circ= n_\ce{i}RT\ln{\left(\frac{a_\ce{i}}{a_\ce{i}^\circ}\right)}$.

> 2. How does it come, that in one case the activity of the whole product AB is important and in another the single activities of the components of the product?

Equation (1) refers to a *molar* free energy of formation of $\ce{AB}$, whereas (2) refers to the free energy of formation of a solid solution of $n_A$ moles of A and $n_B$ moles of B. Equation (1) refers to combination of A and B at a 1:1 mole ratio, or equivalently reaction to form 1 mole of $\ce{AB}$ from  $\mathrm{n_A=n_B=\frac12}\pu{mol}$. Reaction (2) refers to mixture of A and B at any arbitrary ratio or total number of moles. Therefore equation (2) is more general. 

----

> For the given reaction:
> $$\Delta G = \Delta G^⦵ + RT\ln{\frac{a_\ce{AB}}{a_\ce{A}\cdot a_\ce{B}}} $$
> Since all components are pure solid substances, all activities equal 1 and therefore, $\Delta G = \Delta G^⦵$.

A comment on this: the fallacy here is to assume that $a_i=1$ at equilibrium. Only for the pure reagents and products in their standard states it is strictly required by definition that $a_i=1$.