When using matrices, you are really just solving a system of equations. You can use any technique you know to solve the system of equations that a chemical equation represents. Using matrices is only one technique that may be useful for complicated equations or if you have a calculator/MATLAB on hand. In the given example, it would be easier to solve it directly with guess and check.
Writing the system of equations
In your example
$$\ce{C3H8 + O2 -> CO2 + H2O},$$
write in some variables as coefficients:
$$\ce{a C3H8 + b O2 -> c CO2 + d H2O}.$$
Now, we can write out a system of equations based on the stoichiometry of each element on either side of the equation. In other words, write equations that express the number of each element on the left hand side to the number of each element on the right hand side:
\begin{align}
3a &= c\\
8a &= 2d &\Longleftrightarrow&& 4a &= d\\
2b &= 2c + d
\end{align}
Finally, we have to choose a value for one variable to substitute in. This will be the fourth and final equation. We want integer coefficients here, so we will choose a substitution that gives us all integer coefficients. Otherwise, just multiply your coefficients at the end so that they are all integers. Here, I'll let $a = 1$.
Solving the system
Substitution
Substituting the first two equations into the third gives:
\begin{align}
2b &= 6a + 4a = 10a &\Longleftrightarrow&& b &= 5a
\end{align}
\begin{align}
a &= 1\\
b &= 5\\
c &= 3\\
d &= 4
\end{align}
And the balanced equation is
$$\ce{C3H8 + 5O2 -> 3CO2 + 4H2O}.$$
Using matrices
To answer the question though, let's do it with matrices. I won't go through the whole explanation of solving the system but I'll show you how to build the matrices and get an answer. Remember, we are just representing the same system of equations from before, but with matrices now!
We need to build two matrices, $A$ and $b$. Reorganize the equations we wrote previously so that all of the variables are on the left hand side and a constant is on the right hand side. Also, notice I added in our equation let $a = 1$:
\begin{align}
1a + 0b + 0c + 0d &= 1\\
3a + 0b -1c + 0d &= 0\\
0a + 2b -2c - 1d &= 0\\
4a + 0b + 0c - 1d &= 0
\end{align}
These equations are written like this:
\begin{align}
A &=
\begin{bmatrix}
1 & 0 & 0 & 0\\
3 & 0 & -1 & 0\\
0 & 2 & -2 & -1\\
4 & 0 & 0 & -1\\
\end{bmatrix}\\
b &=
\begin{bmatrix}
1\\
0\\
0\\
0\\
\end{bmatrix}
\end{align}
Notice that $b$ is just a column of the constants on the right hand sides of the equations, in the same order as they were given. The order is important here!
There are a few ways to solve this manually or in a calculator/MATLAB. If you have a TI-8x calculator or similar, you could use its rref function (this transforms an augmented matrix into reduced row echelon form, I'm sure google could help you out) or you could multiply $A^{-1}b = x$. In MATLAB, just do: A\b. This gave me:
$$
x =
\begin{bmatrix}
1 \\ 5 \\ 3 \\ 4
\end{bmatrix}
$$
Each row in this final matrix $x$ is the value for $a$, $b$, $c$, $d$ in that order, giving the balanced chemical equation
$$\ce{C3H8 + 5O2 -> 3CO2 + 4H2O}.$$
