The answer to this is very deep, and is rooted in mathematics, not physics. I don't think any chemistry textbook seriously explains what an irreducible representation "really" is, precisely because it's almost impossible to teach it to the average chemist. To properly understand this, I'd suggest reading about group theory, and then representation theory if you're interested.
However, let's see if I can try to provide a brief-ish overview. As will become very clear, covering this in full is not feasible in a Stack Exchange answer. Also, you will likely need some linear algebra background to understand this. I simply can't explain everything from the ground up, so apologies to any readers left behind.
We start from the definition of a group. This is a set of objects $G$, equipped with a binary operation $\cdot$, which obeys the four group axioms, namely:
- closure (for all $g, h \in G$, $g\cdot h \in G$)
- associativity (for all $g, h, k \in G$, $(g \cdot h) \cdot k = g \cdot (h \cdot k)$)
- identity (there exists some element $e \in G$ such that for all $g \in G$, $e\cdot g = g \cdot e = g$)
- inverse (for all $g \in G$, there exists some element $g^{-1} \in G$ such that $g\cdot g^{-1} = g^{-1} \cdot g = e$)
In molecular symmetry, the objects $g$ are symmetry operations. That is, the point group consists of all symmetry operations which preserve the molecular geometry. These elements form the top row of the character table.*
Often in (basic) group theory we are concerned with the elements of the group $g$, i.e. the symmetry operations. However, in chemistry we are really more interested in the molecules or the orbitals, which are not elements of the group! These are actually objects which the group elements act on.
Mathematically speaking, these objects are vectors which inhabit a vector space $V$. For example, in ammonia, we might have a vector space which consists of the three hydrogen 1s orbitals, as well as the 2s and 2p orbitals on nitrogen. It is left as an exercise to the reader to convince themselves that the group elements (i.e. symmetry operations) convert an element of the vector space (e.g. a hydrogen 1s) to another element of the same vector space (e.g. a rotation converts it to the 1s orbital of a different hydrogen).
Vector spaces are defined by a few more axioms, which can be easily found in any reference. Each group element $g_i \in G$ is therefore a map: it acts on a vector $v \in V$ and produces another vector $w \in V$. Of course, this map is different for every group element. So each group element $g_i$ is a different map $\rho_i: V \to V$. The act of "going from group elements $g_i$ to vector mappings $\rho_i$" is called a group homomorphism, or a representation. $V$ is called the representation space. There is a condition on this, namely that all of the $\rho_i$'s are linear:
$$\begin{align}
\rho_i(v_1 + v_2) &= \rho_i(v_1) + \rho_i(v_2) \\
\rho_i(kv_1) &= k\rho_i(v_1)
\end{align}$$
where $v_1, v_2 \in V$ and $k$ is some complex number.†
A different representation simply means a different set of $\rho_i$'s, which may well act on a different vector space $V'$.
If our vector space has finite dimensions, let's say $n$ dimensions, then each vector $v \in V$ can be associated with a $n \times 1$ (complex-valued) column vector. The exact numbers in the column vector depends on the choice of basis. (Again, I can't explain all of this here; this is basic linear algebra.) Then, the fact that the $\rho_i$'s are linear maps on $V$, means that the $\rho_i$'s are actually $n \times n$ matrices. Again, the matrix elements will depend on the basis chosen for $V$. Often you will read that these matrices are elements of the general linear group $GL(n; \mathbb{C})$. Maths textbooks don't like to choose an explicit basis for $V$, but it's necessary for our purposes here.
Let's take stock of where we are. We've explained how group elements may be represented by $n \times n$ complex matrices (assuming a finite-dimensional representation space). These matrix elements $\rho_{i,pq}$ depend on the basis chosen.
$$g_i \to \rho_i = \begin{pmatrix}
\rho_{i,11} & \rho_{i,12} & \cdots & \rho_{i,1n} \\
\rho_{i,21} & \rho_{i,22} & \cdots & \rho_{i,2n} \\
\vdots & \vdots & \ddots & \vdots \\
\rho_{i,n1} & \rho_{i,n2} & \cdots & \rho_{1,nn} \\
\end{pmatrix}$$
The act of decomposing a representation refers to finding a basis of $V$ which make all $\rho_i$'s have (the same) block diagonal form. Mathematicians don't like to speak about bases of $V$, and so this is often framed in terms of "subspaces" and "quotient spaces"; you may see that in books. But we will stick with our explicit matrices and bases.
Let's talk about a three-dimensional basis for now; so the $\rho_i$'s are $3 \times 3$ matrices. (Essentially, $V = \mathbb{C}^3$.) And let's suppose that we find a basis in which
$$g_i \to \rho_i = \begin{pmatrix}
\rho_{i,11} & 0 & 0 \\
0 & \rho_{i,22} & \rho_{i,23} \\
0 & \rho_{i,32} & \rho_{i,33} \\
\end{pmatrix}$$
and that every matrix $\rho_i$, for all group elements $g_i$, has the same block-diagonal form in this basis. Essentially, what a block-diagonal matrix does is it disentangles different basis vectors from one another. If the matrix $\rho_i$ acts on the vector $v_1 = (a, 0, 0)^T$, then we can guarantee that the resulting vector $\rho_i v_1$ is of the form $(a', 0, 0)^T$ (and furthermore $a' = \rho_{i,11}a$). Likewise, if we take $\rho_i$ and act on some other vector $v_2 = (0, b, c)^T$, then the resulting vector $\rho_i v_2$ is $(0, b', c')^T$ where the first element is still $0$.
This means that we have essentially obtained two different representations from this.
The first representation is defined by the vector space $V_1 = \{(a, 0, 0)^T\}$ for some $a \in \mathbb{C}$. We can get rid of the two zero entries such that $V_1$ is basically a 1-dimensional space.
- The $\rho_i$'s in this representation are then just $1 \times 1$ matrices with the single element $\rho_{i,11}$.
The second representation is defined by the vector space $V_2 = \{(0, b, c)^T\}$ for $b, c \in \mathbb{C}$. Again, we can drop the zero entry at the beginning.
- $\rho_i$ in this representation is then the bottom-right $2\times 2$ block of the previous matrix.
Note that the old $3 \times 3$ $\rho_i$ matrix can be obtained through a direct sum of the $1 \times 1$ and $2 \times 2$ matrices. This is where the direct sum comes in.
Alternatively, you could leave in all the zeroes that we took out; and instead of the direct sum we would just have the ordinary sum of two matrices.
These two new representations are called subrepresentations. An irreducible representation is then a subrepresentation which cannot be reduced further in this manner. It remains to make a few points:
- The SALCs that you see are the basis vectors of $V$ which allow for this decomposition.
- The character listed for each irrep in the character table refers to the trace of the matrix $\rho_i$. Note that each group element corresponds to a different matrix with (in general) a different trace, so the character of an irrep under each symmetry operation is (in general) not the same.
- If an irrep is 2-dimensional (for example), then the basis in this 2-dimensional vector space can be arbitrarily chosen (the fact is that it can't be further decomposed into block-diagonal form, regardless of basis: so it doesn't matter what the basis is). So, the SALCs for 2- and higher-dimensional irreps are actually arbitrarily chosen. This is true for the $s_2'$ and $s_3'$ irreps in the question: any linear combination of $s_2'$ and $s_3'$ is a valid pair of SALCs.
- Changing the basis within an irrep does not change the characters of the irrep, because the trace of a matrix is invariant under a change of basis.
Footnotes
* Note that in a character table, elements that are conjugate to one another are grouped into the same entry. Two elements $g, h \in G$ are said to be conjugate if there exists some element $t \in G$ such that $g = tht^{-1}$.
† We are dealing with complex-valued vector spaces. Most of the simple character tables don't have complex numbers. But counterexamples are easy to find: see e.g. the character table for $C_3$. http://symmetry.jacobs-university.de/cgi-bin/group.cgi?group=203&option=4
