Matrix multiplication

Matrix multiplication is a fundamental matrix operation. Modern game engines are built with the help of matrix multiplication, since it's closely related to geometry. Apart from that, it's also used in machine learning, for example, to discover what features of studied objects are the most important. In every subject, where matrices come into play, matrix multiplication plays an important role.

Multiply two matrices

Before multiplying matrices, you should find out whether you can multiply them or not. It may seem odd if you've never worked with matrices, but after some practice, you'll get used to it. So, to define multiplication:

• The number of columns of the $1$st matrix must be equal to the number of rows of the $2$nd matrix.

• The result will have the same number of rows as the $1$st matrix, and the same number of columns as the $2$nd matrix.

Let's move on and multiply matrices $A=\begin{pmatrix} 2 & 4 & 6 \\ 1 & 3 & 5 \end{pmatrix}$and $B=\begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix}$ To do this, you need to know the dot product of their rows and columns. You've already learned about vector dot product and how to calculate it. Now you're going to deal with the same thing but for matrices. In case of matrices, a dot product is a sum of the products of corresponding elements from a row of the first matrix and a column of the second, for example, the 1st row and the 1st column:

$\begin{pmatrix} {\color{red}2} & {\color{red}4} & {\color{red}6} \\ 1 & 3 & 5 \end{pmatrix} \cdot \begin{pmatrix} {\color{blue}1} & 2 \\ {\color{blue}3} & 4 \\ {\color{blue}5} & 6 \end{pmatrix} = \begin{pmatrix} {\color{red}2} \cdot {\color{blue}1} + {\color{red}4} \cdot {\color{blue}3} +{\color{red}6} \cdot {\color{blue}5} & \\ & \end{pmatrix} = \begin{pmatrix} {\color{green}44} & \\ & \end{pmatrix}$

$44$ is the dot product of the $1$st row and the $1$st column.

What is the dot product of the $1$st row and the $2$nd column?

$\begin{pmatrix} {\color{red}2} & {\color{red}4} & {\color{red}6} \\ 1 & 3 & 5 \end{pmatrix} \cdot \begin{pmatrix} 1 & {\color{blue}2} \\ 3 & {\color{blue}4} \\ 5 & {\color{blue}6} \end{pmatrix} = \begin{pmatrix}44 & {\color{red}2} \cdot {\color{blue}2} + {\color{red}4} \cdot {\color{blue}4} +{\color{red}6} \cdot {\color{blue}6}\\ & \end{pmatrix} = \begin{pmatrix} 44 & {\color{green}56} \\ & \end{pmatrix}$

It is $56$.

What about the dot product of the $2$nd row and the $1$st column?

$\begin{pmatrix} {\color{red}1}, {\color{red}3}, {\color{red}5}\end{pmatrix} \cdot \begin{pmatrix} {\color{blue}1},{\color{blue}3}, {\color{blue}5} \end{pmatrix} = \begin{pmatrix} {\color{red}1} \cdot {\color{blue}1} + {\color{red}3} \cdot {\color{blue}3} +{\color{red}5} \cdot {\color{blue}5} \end{pmatrix} = 35$

In the same way, the dot product of the $2$nd row and the $2$nd column equals $44$. So the result of the multiplication of matrices $A$ and $B$ will be:

$\begin{pmatrix}2 & 4 & 6 \\ 1 & 3 & 5 \end{pmatrix} \cdot \begin{pmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{pmatrix} = \begin{pmatrix} 44 & 56 \\ 35 & 44 \end{pmatrix}$

Such things might be hard to remember, so let's look at a real-life example.

Example

A local shop sells three types of useful things:

• AR glasses cost $\$ 5$ each

• Hoverboards cost $\$ 5$ each

• USB pet rocks cost $\$ 10$ each

This table shows how many items were sold in four days:

The value of sales for Tuesday is calculated this way: AR glasses Tuesday value $+$ Hoverboard Tuesday value $+$ USB pet rock Tuesday value

So it is, in fact, the dot product of the prices and the number of sold items on Tuesday:

$$(\$5,\$5,\$10)\cdot(5,7,15)=\$5\cdot5+\$5\cdot7+\$10\cdot15 = {\color{red}\$210}$$

We match the price to the number of sold items, multiply each, then sum up the result.

We do the same for other days:

Wed = $\$5\cdot4+\$5\cdot5+\$10\cdot20 ={\color{red}\$245}$

Thu = $\$5\cdot6+\$5\cdot3+\$10\cdot25 ={\color{red}\$295}$

Fri = $\$5\cdot5+\$5\cdot6+\$10\cdot20 ={\color{red}\$255}$

So what just happened here in terms of matrices?

$$\begin{pmatrix} \$5 & \$5 & \$10 \end{pmatrix} \cdot \begin{pmatrix} 5 & 4 & 6 & 5 \\ 7& 5 & 3 & 6 \\ 15 & 20 & 25 & 20 \end{pmatrix} = \begin{pmatrix} \$210 & \$245 & \$295 & \$255 \end{pmatrix}$$

To multiply ${\color{red}m} \times {\color{green}a}$ matrix by ${\color{green}a} \times {\color{blue}n}$ matrix, ${\color{green}a}$ must be the same, and the result is ${\color{red}m} \times {\color{blue}n}$ matrix. In our case we did $A_{1,3} \cdot B_{3,4} = C_{1,4}$

Order of multiplication

What about other matrices? Let's multiply the matrices by each other and check it out.

$\begin{pmatrix} 3 & 1 \\ 0 & 2 \end{pmatrix} \cdot \begin{pmatrix} 1 & 2 \\ -1 & 2 \end{pmatrix} = \begin{pmatrix} 3\cdot 1 + 1 \cdot (-1) & 3 \cdot2+1\cdot 2 \\ 0\cdot 1+ 2\cdot (-1)& 0 \cdot 2 +2 \cdot 2 \end{pmatrix} = \begin{pmatrix} 2 & 8 \\ -2& 4 \end{pmatrix}$

If we swap these matrices and multiply again, what happens then?

$\begin{pmatrix} 1 & 2 \\ -1 & 2 \end{pmatrix} \cdot \begin{pmatrix} 3 & 1 \\ 0 & 2 \end{pmatrix} = \begin{pmatrix} 1\cdot 3 + 2 \cdot 0 & 1 \cdot1+2\cdot 2 \\ -1\cdot 3+ 2\cdot 0& -1 \cdot 1 +2 \cdot 2 \end{pmatrix} = \begin{pmatrix} 3 & 5 \\ -3& 3 \end{pmatrix}$

Wow! The results are different! Forget about the commutative law in multiplication: it usually does not work for matrices. Even if you're used to $3 \cdot 5 = 5 \cdot 3$, in matrix notation $AB \ne BA$. The order is very important for the multiplication of matrices. There may be some cases when the commutative law works, but they're quite unique. We'll discuss one of them later.

Multiplication by a diagonal or a scalar matrix

Let's consider two special cases of matrix multiplication: multiplication by a diagonal and a scalar matrix.

Multiplication by a diagonal matrix multiplies all rows by corresponding leading entries of the diagonal matrix:

$\begin{pmatrix} \color{red}{1} & 0 & 0 \\ 0 & \color{red}{3} & 0 \\ 0 & 0 & \color{red}{7} \end{pmatrix} \cdot \begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{pmatrix} = \begin{pmatrix}\color{red}{1} \color{black} \cdot 1 & \color{red}{1} \color{black} \cdot 2 & \color{red}{1} \color{black} \cdot 3 \\ \color{red}{3} \color{black} \cdot 4 & \color{red}{3} \color{black} \cdot 5 & \color{red}{3} \color{black} \cdot 6 \\ \color{red}{7} \color{black} \cdot 7 & \color{red}{7} \color{black} \cdot 8 & \color{red}{7} \color{black} \cdot 9 \end{pmatrix} = \begin{pmatrix} 1 & 2 & 3 \\ 12 & 15 & 18 \\ 49 & 56 & 63 \end{pmatrix}$

Why is it so? The dot product of the $1$st row of the diagonal matrix and the $1$st column of the right matrix, the only non-zero element is equal to $1$: $(\red{1}, 0, 0)\cdot(1,4,7)=1\cdot1+0\cdot4+0\cdot7=1$

The same thing happens with all other dot products calculated during matrix multiplication.

Right multiplication by a diagonal matrix multiplies all columns by corresponding leading entries of the diagonal matrix:

$\begin{pmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \\ 7 & 8 & 9 \end{pmatrix} \cdot \begin{pmatrix} \color{red}{1} & 0 & 0 \\ 0 & \color{red}{3} & 0 \\ 0 & 0 & \color{red}{7} \end{pmatrix} = \begin{pmatrix}\color{red}{1} \color{black} \cdot 1 & \color{red}{3} \color{black} \cdot 2 & \color{red}{7} \color{black} \cdot 3 \\ \color{red}{1} \color{black} \cdot 4 & \color{red}{3} \color{black} \cdot 5 & \color{red}{7} \color{black} \cdot 6 \\ \color{red}{1} \color{black} \cdot 7 & \color{red}{3} \color{black} \cdot 8 & \color{red}{7} \color{black} \cdot 9 \end{pmatrix} = \begin{pmatrix} 1 & 6 & 21 \\ 4 & 15 & 42 \\ 7 & 24 & 63 \end{pmatrix}$The explanation here is nearly the same as in the previous case. The dot product of the $1$st row of the left matrix and the $1$st column of the diagonal matrix, the only non-zero element is equal to $1$: $(1, 2, 3)\cdot(\red{1},0,0)=1\cdot1+2\cdot0+3\cdot0=1$

Here, the same thing also happens with all other dot products calculated during matrix multiplication.

And what about a scalar matrix? Since it's a diagonal matrix in which all the principal diagonal elements are equal, multiplication by it would multiply all rows or columns by the same value. Therefore, the resulting matrix would be a matrix, all entries of which are equal to the entries of the original matrix multiplied by a scalar, hence the name.

Multiply matrix and vector

A vector is an $n\times1$ matrix, where $n$ is a natural number, so it is also a case of matrix multiplication. Usually, you use a $2$ or $3$-dimensional space, so $n$ in most cases equals $2$ or $3$.

Let's consider that you use a $3$-dimensional space and have a $2\times 3$ matrix $A=\begin{pmatrix} 1 & -1 & 0\\ 2 & 0 & 1\end{pmatrix}$ and a vector $x = \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix}$. Here, the number of columns of the matrix $A$ is equal to the number of rows of the vector $x$, so you can find their product. $A\cdot x = \begin{pmatrix} 1 & -1 & 0\\ 2 & 0 & 1\end{pmatrix} \cdot \begin{pmatrix} 1 \\ 1 \\ 1 \end{pmatrix} = \begin{pmatrix} 1\cdot1-1\cdot1+0\cdot1\\ 2\cdot 1+0\cdot1 + 1\cdot1\end{pmatrix} = \begin{pmatrix} 0\\ 3\end{pmatrix}$

In this case, the result is a $2\times 1$ vector. In general, if you multiply an $m\times n$ matrix by an $n\times 1$ vector, you'll always get an $m\times1$ vector.

Can you change the order and multiply a vector by a matrix? Every vector has only one column, so in order to be able to multiply a vector by a matrix, you need a $1\times m$ matrix where $m$ is a natural number. You can obtain this matrix by transposing an $m\times 1$ vector! So, you can multiply a vector only by another transposed vector, but not by any matrix. Here's an illustration of such multiplication.

Identity matrix & Commutative law

Do you remember the rule from mathematics $A \cdot 1 = A$? Is there such a rule for matrices? Sure! It is the Identity matrix denoted as $I$. When we multiply a matrix by the Identity matrix, the commutative law applies:

$A \cdot I = I \cdot A = A$

Check it out!

$\begin{pmatrix} 2 & 4 \\ 6 & 8 \end{pmatrix} \cdot \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} \cdot \begin{pmatrix} 2 & 4 \\ 6 & 8 \end{pmatrix} = \begin{pmatrix} 2 & 4 \\ 6 & 8 \end{pmatrix}$

Matrix multiplication from childhood

Matrix multiplication is a beautiful mathematical process. Remember the multiplication table from school?

In fact, in matrix form, it looks like:

$\begin{pmatrix}1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 \end{pmatrix}^T \cdot \begin{pmatrix}1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 \end{pmatrix} \\ = \begin{pmatrix} 1 & 2 & 3 &4 & 5 &6 &7 &8 &9 \\ 2 & 4 &6 & 8 & 10 & 12 & 14 & 16 & 18 \\ 3 & 6 & 9 & 12 &15 &18 & 21 & 24 & 27 \\ 4 & 8 & 12 & 16 & 20 & 24 & 28 &32 & 36 \\ 5 & 10 &15 &20 & 25 & 30 & 35 &40 & 45 \\ 6 & 12 &18 &24 & 30 & 36 & 42 & 48 & 54 \\ 7 & 14 & 21 & 28 & 35 & 42 & 49 & 56 & 63 \\ 8 & 16 & 24 & 32 & 40 & 48 & 56 & 64 & 72 \\ 9 & 18 & 27 & 36 & 45 & 54 & 63 & 72 & 81 \end{pmatrix}$

Don't believe it? Now you have enough knowledge to verify that.

Multiplying many matrices

How to multiply more than two matrices? Imagine that you have three matrices $A$, $B$ and $C$. In order to find $A\cdot B \cdot C$, you have to multiply $A$ by $B$ and after that, multiply the result by $C$.

Let's consider an example. Let $A =\begin{pmatrix} 1&1&1 \end{pmatrix}$, $B=\begin{pmatrix} 0&0 \\ 1&2 \\ 3 & 4\end{pmatrix}$ and $C=\begin{pmatrix} 5 \\ 0 \end{pmatrix}$. In this case $A\cdot B = \begin{pmatrix} 1&1&1 \end{pmatrix} \cdot \begin{pmatrix} 0&0 \\ 1&2 \\ 3 & 4\end{pmatrix}= \begin{pmatrix}4 & 6 \end{pmatrix}$

And the final result is$A\cdot B \cdot C =\begin{pmatrix}4 & 6 \end{pmatrix} \cdot \begin{pmatrix} 5 \\ 0 \end{pmatrix} = \begin{pmatrix} 20 \end{pmatrix}$

Conclusion

Let's summarize what we have learned today, all this knowledge is fundamental when working with matrices in further topics:

• Dot product represents the addition of corresponding elements from a row of a first matrix with the corresponding element of a column of a second matrix.

• For us to be able to multiply matrices, the number of columns of the first matrix has to be equal to the number of rows of the second matrix.

• When multiplying two matrices, the resulting matrix will have the same number of rows as the first matrix and the same number of columns as the second one.

• Commutative law does not usually apply when multiplying matrices. If you multiply a matrix $A$ by matrix $B$, you will not always get the same matrix as from multiplying matrix $B$ by matrix $A$.