Spectral decomposition

Diagonalization is the fundamental matrix decomposition. From this topic, you will begin to know improved versions of it that will bring you closer to the definitive understanding of all the properties of any matrix.

Today you will start with the spectral decomposition. The good news is that it's easy to compute from diagonalization. With it, you'll understand all the geometry of symmetric matrices, and you'll put the properties of unitary matrices to work.

An improved decomposition

As you know, a square matrix $A$ of dimension $n$ is diagonalizable if there exists a diagonal matrix $D$ and an invertible one $P$, both of size $n$, such that $A=PDP^{−1}$. Since the diagonal entries of $D$ are the eigenvalues of $A$, while the columns of $P$ are the eigenvectors of $A$, this result is known as an eigendecomposition of $A$.

Building this decomposition isn't easy. Although calculating all the eigenvalues and eigenvectors of $A$ can be relatively straightforward, next comes a perhaps more complicated task: inverting the matrix $P$. Since its columns form a basis, the inverse always exists, but calculating it can involve a lot of work. Fortunately, there's a class of matrices whose inverses are trivial to calculate: unitary matrices, since their inverses are just their transposes. So, it is natural to ask when the invertible matrix $P$ can be changed to a unitary one.

So, as you remember, to build $P$ you just need to compute a basis for each eigenspace and then combine them all to form a basis for the whole space. But wait just a minute. The columns of an orthogonal matrix form an orthonormal basis. Would improving the basis of each eigenspace by an orthonormal one be enough? After that, you'd only need to put them all together so that you get the orthonormal basis for the entire space. Finally, by putting their elements as columns of a matrix, this would end up unitary.

There is a small problem with this procedure. It's true that every eigenspace has an orthonormal basis. It is also true that by putting together all the orthonormal bases of the eigenspaces, you would end up with a basis for the whole space. It is clear that all basic vectors have lengths equal to $1$. Also, if they belong to the same eigenspace, then they are orthogonal. But nothing guarantees that if they come from different eigenspaces, they're orthogonal to each other! That is, the final base may not be orthogonal. You were very close to success.

But there's still hope. Although symmetric matrices may seem simple at first, it is surprising that they're precisely the only ones for which this improved decomposition works. That is, the eigenvectors of different eigenspaces are orthogonal to each other. With this, you're finally ready for the promised result, the spectral decomposition:

Although symmetric matrices may seem unhelpful at first, they arise naturally in many applications. For example, probability studies the interactions between different random phenomena through the covariance matrix that turns out to be symmetric. In graph theory, which studies relationships between similar objects (for example, you might have a set of cities that you want to connect by building roads), the adjacency matrix is used to order these relationships, which also ends up being symmetric.

The geometry of symmetric matrices

Thanks to spectral decomposition, you can decompose the geometric behavior of any symmetric matrix $A$ in simple steps. First, notice that the linear operator of $A$ is given by the following composition of more understandable ones:$$L_A=L_U \circ L_D \circ L_{U^T}$$So, the behavior of $L_A$ is described by three stages:

• As $U^T$ is unitary, the action of $L_{U^T}$ is just a rotation of the space,
• since $D$ is diagonal, it implies that $L_D$ only stretches the space,
• finally $L_{U}$ is a rotation in the inverse direction that one of $L_{U^T}$.

Note that the first rotation is countered in the third stage, so essentially $L_A$ is very much like a stretch of the whole space.

But, how to actually obtain such a decomposition? Let's get to it!

Computing the decomposition

The procedure to calculate the spectral decomposition is almost identical to the one you already know to find the diagonalization. You just need one extra step that is quite predictable.

1. Find all the eigenvalues of $A$.

2. Compute the eigenspace of every eigenvalue.

3. Form an orthonormal basis for every eigenspace.

4. Build $U$: its columns are the elements of all the orthonormal bases.

5. Build $D$: the entries on its diagonal are the eigenvalues corresponding to the eigenvectors in the same order that in $U$. If an eigenvalue has more than one eigenvector, then it is repeated as many times as many eigenvectors it has in $U$.

Here is a pair of clarifications that will make it easier to calculate the decomposition:

• By the spectral theorem, every symmetric matrix is automatically diagonalizable, so you don't need to worry about the geometric dimension of the eigenvalues.
• An eigenspace is a subspace, so you can compute a basis for it. From that basis, you can build an orthonormal basis—for example, use the Gram-Schmidt procedure.

Let's compute a couple of decompositions.

A quick example

Consider the following matrix:

$$A = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix}$$The first step is computing all its eigenvalues. Calculating its characteristic polynomial $\chi(\lambda) = \det(A-\lambda I) = \lambda^2 - 6 \lambda + 5$, you can notice that the eigenvalues are $\textcolor{black}{5}$ and $\textcolor{purple}{1}$. After that, you must find the eigenspaces and form an orthonormal basis for each one:

• $\textcolor{teal}{5}$: its eigenspace is $\textcolor{black}{E_5} = \{x \ (1,1)^T \, : x \in \mathbb{R} \}$. Hence a basis is simply $\{ \textcolor{black}{(1,1)^T} \}$ but the extra step requires an orthonormal basis, so you can use $\left\{\textcolor{black}{\frac{1}{\sqrt{2}} \ (1,1)^T} \right\}$.
• $\textcolor{purple}{1}$: its eigenspace is $\textcolor{black}{E_1} = \{x \ (-1,1)^T \, : x \in \mathbb{R} \}$ and an orthonormal basis is $\left\{ \textcolor{black}{ \frac{1}{\sqrt{2}} \ (-1,1)^T} \right\}$.

The final steps are building $U$ and $D$ using the bases:

$$U = \frac{1}{\sqrt{2}} \begin{pmatrix} \textcolor{teal}{1} & \textcolor{purple}{-1} \\ \textcolor{teal}{1} & \textcolor{purple}{1} \end{pmatrix} \qquad D = \begin{pmatrix} \textcolor{teal}{5} & 0 \\ 0 & \textcolor{purple}{1} \end{pmatrix}$$Thus the spectral decomposition is $A=U D U^T$.

Being careful

Now take a bigger matrix:

$$A = \begin{pmatrix} -1 & 2 & 2 \\ 2 & -1 & 2 \\ 2 & 2 & -1\end{pmatrix}$$It's easy to compute its characteristic polynomial, $\chi(\lambda) = \det(A-\lambda I) = -\lambda^3 -3 \lambda^2 +9 \lambda + 27$, so the eigenvalues are $\textcolor{black}{3}$ and $\textcolor{purple}{-3}$.

• $\textcolor{black}{3}$: the eigenspace is $\textcolor{black}{E_3} = \{x \ (1,1, 1)^T : x \in \mathbb{R} \}$. In consequence, an orthonormal basis is $\{ \textcolor{black}{ \frac{1}{\sqrt{3}} (1, 1,1)^T} \}$.
• $\textcolor{black}{-3}$: now the eigenspace is $\textcolor{black}{E_{-3}} = \{x \ (-1, 0, 1)^T +y \ (-1,1,0)^T \, : x, y \in \mathbb{R} \}$. This case is more delicate because the basis $\{ \ (-1, 0, 1)^T, \ \ (-1,1,0)^T \}$ isn't orthogonal. But you can use the Gram-Schmidt procedure to build an orthonormal basis, for example, $\{\textcolor{black}{ \frac{1}{\sqrt{2}} (-1, 0, 1)^T, \ \ \frac{1}{\sqrt{2}}(- \frac{1}{\sqrt{3}}, \frac{2}{\sqrt{3}}}, \textcolor{black}{ - \frac{1}{\sqrt{3}} )^T } \}$.

Great, then the pieces of the decomposition are:

$$U = \frac{1}{\sqrt{2}} \begin{pmatrix} \textcolor{black}{\sqrt{\frac{2}{3}}} & \textcolor{black}{-1} & \textcolor{black}{\frac{-1}{\sqrt{3}}} \\ \textcolor{black}{\sqrt{\frac{2}{3}}} & \textcolor{black}{0} & \textcolor{black}{\frac{2}{\sqrt{3}}} \\ \textcolor{black}{ \sqrt{\frac{2}{3}}} & \textcolor{black}{1} & \textcolor{black}{\frac{-1}{\sqrt{3}}} \end{pmatrix} \qquad D = \begin{pmatrix} \textcolor{black}{3} & 0 & 0 \\ 0 & \textcolor{black}{-3} & 0 \\ 0 & 0 & \textcolor{black}{-3} \end{pmatrix}$$

Conclusion

• The spectral decomposition is an improved diagonalization that characterizes symmetric matrices.

• A matrix $A$ is symmetric if and only if there exists a unitary matrix $U$ and a diagonal one $D$, both of size $n$, such that $A=U D U^T$.

• The geometric behavior of a symmetric matrix is the composition of a rotation, a stretching, and a rotation in the opposite direction.

• In order to compute the spectral decomposition, you just have to diagonalize the matrix but with the slight modification that the eigenvectors must be orthogonal and of length $1$.