Chapter 14 Eigenvalues and Eigenvectors

In this chapter, you will learn about eigenvalues and eigenvectors. Eigenvalues (a.k.a., characteristic roots) are scalars that are associated with linear systems of equations. Each eigenvalue has a corresponding vector, an eigenvector, associated with it. Eigenvalues and eigenvectors play a role in many statistical applications, featuring prominently in some methods of decomposition and principal component analysis (PCA).

14.1 Eigenvalues

Let A be a \(p \times p\) square matrix and I be a \(p \times p\) identity matrix. Eigenvalues are the set of non-zero scalars \(\lambda_1,\lambda_2, \lambda_3,\ldots,\lambda_p\) that satisfy,

\[ \mathrm{det}\bigg(\mathbf{A} - \lambda \mathbf{I}\bigg) = 0 \]

Consider an example,

\[ \underset{2\times 2}{\mathbf{A}} = \begin{bmatrix} -3 & 5 \\ 4 & -2 \\ \end{bmatrix} \]

Our goal is to find the values of \(\lambda\) that satisfy

\[ \mathrm{det}\bigg(\begin{bmatrix} -3 & 5 \\ 4 & -2 \\ \end{bmatrix} - \lambda\begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}\bigg) = 0 \]

Using matrix algebra,

\[ \begin{split} \mathrm{det}\bigg(\begin{bmatrix} -3 & 5 \\ 4 & -2 \\ \end{bmatrix} - \lambda\begin{bmatrix} 1 & 0 \\ 0 & 1 \\ \end{bmatrix}\bigg) &= 0 \\[2ex] \mathrm{det}\bigg(\begin{bmatrix} -3 & 5 \\ 4 & -2 \\ \end{bmatrix} - \begin{bmatrix} \lambda & 0 \\ 0 & \lambda \\ \end{bmatrix}\bigg) &= 0 \\[2ex] \mathrm{det}\begin{bmatrix} -3 - \lambda & 5 \\ 4 & -2 - \lambda \\ \end{bmatrix} &= 0 \\[2ex] (-3 - \lambda)(-2 - \lambda) - 20 &= 0 \end{split} \]

Computing the determinant, distributing the first set of terms, subtracting 20 we obtain:

\[ \begin{split} (-3 - \lambda)(-2 - \lambda) - 20 &= 0 \\[1em] 6 + 3\lambda + 2\lambda + \lambda^2 -20 &= 0 \\[1em] \lambda^2 + 5\lambda - 14 &= 0 \\[1em] \end{split} \]

Factoring the left-hand side,

\[ (\lambda + 7)(\lambda - 2) = 0 \]

Solving for \(\lambda\), we find that \(\lambda = -7\) and \(\lambda = 2\). We could also have applied the quadratic formula to solve this for \(\lambda\).

Remember the quadratic formula? The roots of the polynomial \(Ax^2+Bx+C\) are computed using:

\[ x = \frac{-B\pm \sqrt{B^2-4AC}}{2A} \]

Quick recap at Khan Academy if you need it.

Now that we have the eigenvalues we can double-check that \(\mathbf{A} - \lambda \mathbf{I}\) is singular. (I will skip this here, but plug in the values for \(\lambda\), one at a time, and ensure that the determinant is zero.) The equation you solved,

\[ \mathrm{det}\big(\mathbf{A} - \lambda \mathbf{I}\big)=0 \]

is referred to as the characteristic equation. Solving the characteristic equation gives the eigenvalues. Sometimes the eigenvalues are referred to as the characteristic roots of matrix A.

14.2 Eigenvectors

If \(\lambda\) is an eigenvalue of matrix A, then it is possible to find a vector v (an eigenvector) that satisfies

\[ \mathbf{A}\mathbf{v} = \lambda \mathbf{v} \]

In our previous example, A was a \(2\times2\) matrix, so v will be a \(2\times1\) vector to make the matrix multiplication work.

\[ \underset{2\times2}{\mathbf{A}}~\underset{2\times1}{\mathbf{v}} = \lambda \underset{2\times1}{\mathbf{v}} \]

Notice that we can also re-arrange the terms of this expression:

\[ \begin{split} \mathbf{A}\mathbf{v} &= \lambda \mathbf{v} \\[2ex] \mathbf{A}\mathbf{v} - \lambda \mathbf{v} &= 0\\[2ex] \big(\mathbf{A} - \lambda \mathbf{I}\big) \mathbf{v} &= 0\\[2ex] \end{split} \]

In order to obtain a nonzero solution for v, the determinant of \(\mathbf{A} - \lambda \mathbf{I}\) needs to be zero. This is why the eigenvalues are the \(\lambda\)-values that make this nonsingular.

14.2.1 Finding the Elements of v

We can use matrix algebra to solve for the elements of vector v using each eigenvalue seperately.

\[ \begin{split} \mathbf{A}\mathbf{v} &= \lambda \mathbf{v} \\[1em] \begin{bmatrix} -3 & 5 \\ 4 & -2 \\ \end{bmatrix}\begin{bmatrix} v_1 \\ v_2 \\ \end{bmatrix} &= -7\begin{bmatrix} v_1 \\ v_2 \\ \end{bmatrix}\\[1em] \end{split} \]

This produces a system of two equations with two unknowns:

\[ \begin{split} -3v_1 + 5v_2 &= -7v_1 \\ 4v_1 - 2v_2 &= -7v_2 \end{split} \]

Simplifying this, we get

\[ \begin{split} 4v_1 + 5v_2 &= 0 \\ 4v_1 + 5v_2 &= 0 \end{split} \]

The homogeneous set of equations means that there are an infinite number of solutions. The general solution here is to express one variable (say \(v_2\)) as a function of the other.

\[ \begin{split} v_1 &= \theta \\[0.5em] v_2 &= -\frac{4}{5}\theta \end{split} \]

Any set of \(v_1\) and \(v_2\) in which \(v_2 = -\frac{4}{5}v_1\) will satisfy this set of equations.

14.2.2 Normalized Eigenvectors

Although there are an infinite number of solutions, one that is particularly nice is that whose a sum of squared values is equal to 1, that is \(\mathbf{v}^\intercal\mathbf{v}=1\). This is referred to as normalizing the eigenvector. Normalized eigenvectors are denoted as e.

\[ \begin{split} \mathbf{e}^\intercal \mathbf{e} &= 1 \\[2ex] e_1^2 + e_2^2 &= 1 \\[2ex] \theta^2 + (-\frac{4}{5}\theta)^2 &= 1 \\[2ex] \frac{41}{25}\theta^2 &= 1 \\[2ex] \theta^2 &= \frac{25}{41} \\[2ex] \theta &= \sqrt{\frac{25}{41}} \\[2ex] &=\frac{5}{\sqrt{41}} \end{split} \]

Which implies that,

\[ \begin{split} e_1 &= \frac{5}{\sqrt{41}} \\[1em] e_2 &= -\frac{4}{\sqrt{41}} \end{split} \]

And the normalized eigenvector corresponding to the eigenvalue of \(-7\) is:

\[ \mathbf{e}_1x = \begin{bmatrix}\frac{5}{\sqrt{41}} \\ -\frac{4}{\sqrt{41}}\end{bmatrix} \]

We could verify this by ensuring that the the equation \(\mathbf{A}\mathbf{e}=\lambda\mathbf{e}\) holds:

\[ \begin{bmatrix} -3 & 5 \\ 4 & -2 \\ \end{bmatrix}\begin{bmatrix}\frac{5}{\sqrt{41}} \\ -\frac{4}{\sqrt{41}}\end{bmatrix} = -7\begin{bmatrix}\frac{5}{\sqrt{41}} \\ -\frac{4}{\sqrt{41}}\end{bmatrix} \]

We can follow the same process for the second eigenvector which corresponds to the eigenvalue of 2. This produces an eigenvector of:

\[ \mathbf{e}_2 = \begin{bmatrix}\frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}}\end{bmatrix} \]

Which can again be verified.

14.3 Using R to Find Eigenvalues and Eigenvectors

We can use the eigen() function to find the eigenvalues and eigenvectors for a matrix. The syntax to find these for our example matrix A is:

# Create A
A = matrix(
  data = c(-3, 4, 5, -2), 
  nrow = 2
  )


# Compute eigenvalues and eigenvectors
eigen(A)

eigen() decomposition
$values
[1] -7  2

$vectors
           [,1]       [,2]
[1,] -0.7808688 -0.7071068
[2,]  0.6246950 -0.7071068

The eigenvectors for each of the eigenvalues are presented in the columns of the matrix given in the `\(vector\) component of the output. For example, the eigenvector associated with the eigenvalue of \(-7\) is presented in the first column of the matrix, and that associated with the eigenvalue of 2 is presented in the second column of the matrix.

Note that the signs for all elements of any particular eigenvector can be switched since the vector would represent the same vector space. Thus the second eigenvector outputted here has negative values.

Together, the eigenvalues and eigenvectors make up the eigenstructure of matrix A. In our example, the eigensructure of A is:

\[ \begin{split} \lambda_1 &= -7 \qquad &\mathbf{e}_1 = \begin{bmatrix}\frac{5}{\sqrt{41}} \\ -\frac{4}{\sqrt{41}}\end{bmatrix}\\[2ex] \lambda_2 &= 2 \qquad &\mathbf{e}_2 = \begin{bmatrix}\frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}}\end{bmatrix} \\[2ex] \end{split} \]