I'd like to describe the symmetry of the ground state of $N_2$ in the higher symmetry groups like $D_{3h}$ and $D_{\infty h}$.
In this answer the topic is covered very good, but only for $D_{2h}$ point group, where $N_2$ orbitals are described by the irreducible representations like this:
$$\begin{array}{cccc} \hline \mathrm{s} & \mathrm{p}_x & \mathrm{p}_y & \mathrm{p}_z \\ \hline \mathrm{A_{g}} & \mathrm{B_{3u}} & \mathrm{B_{2u}} & \mathrm{B_{1u}} \\ \hline \end{array}$$
But, when I have a look at character tables for $D_{3h}$ and $D_{\infty h}$, there are no irreducible representations with basis functions $x$ and $y$ as can be seen below.
So, how am I supposed to describe the symmetry of the $p_x$ and $p_y$ orbitals to be able to represent the symmetry of the whole system subsequently?
My attempt
I've tried to describe the symmetry of $p_x$ in $D_{3h}$ like this:
$$ \begin{array}{cccccc} \hline E & 2C_3(z) & 3C^{'}_{2} & \sigma_h(xy) & 2S_3 & 3\sigma_v\\ \hline 1 & 0 & -1 & 1 & 0 & 0\\ \hline \end{array} $$
Having this reducible representation, I've tried to identify needed irreducible representation using the reduction formula described here:
$$ a_i = \frac{1}{h}\sum_Q N\cdot \chi(R)_Q \cdot \chi_i(R)_Q $$
For the first IR I'm getting the following result
$$ a_{A^{'}_1} = \frac{1}{12}\left[ 1\cdot1\cdot1 + 0 + 3\cdot(-1)\cdot1 + 1\cdot1\cdot1 + 0 + 0 \right] = -\frac{1}{12}, $$
which doesn't maky any sense, as it should be an integer, if I understand it well.
What am I doing wrong here? Is my reducible representation incorrect or does the formula not apply in this situation?
