I'm wondering which atomic orbital is left in a complex with quadruple metal-metal bond to bind the steric ligand, since all d orbitals except the x2-y2 are used for the metal-metal bonds?
or is it the x2-y2 atomic orbital that forms the bonding?
and if so: what atomic orbital is used in metal-metal quintuple complexes?
Here is an example complex: https://en.wikipedia.org/wiki/Quintuple_bond
To determine what orbitals do, it is often a good idea to look at the symmetry group as it can provide a rough overview over what can (symmetry-allowed) and cannot (symmetry-forbidden) happen. In the case of a quadruple bond such as $\ce{[Re2Cl4]^2-}$, we can approximate the rhenium(III) centres as being close to $C_\mathrm{4,v}$. A quick glance at the corresponding group theory table (copied below to avoid page loading issues in the large linked page) will show us which irreducible representations we may need to consider.
$$\begin{array}{c|ccccc|cc} \hline C_\mathrm{4v} & E & 2C_4 & C_2 & 2\sigma_\mathrm{v} & 2\sigma_\mathrm{d} & & \\ \hline \mathrm{A_1} & 1 & 1 & 1 & 1 & 1 & z & x^2+y^2, z^2 \\ \mathrm{A_2} & 1 & 1 & 1 & -1 & -1 & R_z & \\ \mathrm{B_1} & 1 & -1 & 1 & 1 & -1 & & x^2-y^2 \\ \mathrm{B_2} & 1 & -1 & 1 & -1 & 1 & & xy \\ \mathrm{E} & 2 & 0 & -2 & 0 & 0 & (x,y),(R_x,R_y) & (xz,yz) \\ \hline \end{array}$$
The final two columns show the relevant orbitals. The s orbital always transforms as $\mathrm{a_1}$, the p orbitals as $\mathrm{a_1+e}$ and the d orbitals as $\mathrm{a_1+b_1+b_2+e}$. The four chlorine atoms will bond using p orbitals (but the orbital type does not matter as we will only be considering σ bonds). They transform as $\mathrm{a_1+b_1+e}$ (calculation not shown). Immediately, we notice that $\mathrm d_{xy}$ does not contribute to bonding to the ligands but only to the $\ce{Re-Re}$ quadruple bond. Furthermore, we notice that $\mathrm d_{x^2-y^2}$ is practically required to bond to the chlorines as there is no other rhenium orbitals of that symmetry.
To determine what is left, it may be easiest to draw out the potential chlorine orbitals. $\mathrm{a_1}$ means that all phases pointing to rhenium are the same. This can bond either with rhenium’s 6s or $\mathrm{5d}_{z^2}$ orbital or a linear combination of both. The $\mathrm p_z$ orbital does not have any contribution in the $xy$ plane and thus cannot participate.
The chlorine’s $\mathrm e$ orbitals correspond to two in which two opposite chlorines have opposite phases and the other two have zero contribution (in one case, the $x$ chlorines have contribution, in the other the $y$ ones). Thus, to bond with these we will use $\mathrm{6p}_x$ and $\mathrm{6p}_y$. Again, this is because the corresponding d orbitals do not have any contribution in this $xy$ plane.
Thus, to sum things up:
If the symmetry around a rhenium atom is assumed to be $C_4$ instead of $C_\mathrm{4,v}$, that doesn’t change much of the analysis; only the distinction between $\mathrm d_{x^2-y^2}$ and $\mathrm d_{xy}$ becomes irrelevant.
