Section SVD  Singular Value Decomposition

From A First Course in Linear Algebra
Version 1.08
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

This Section is a Draft, Subject to Changes
Needs Numerical Examples

The singular value decomposition is one of the more useful ways to represent any matrix, even rectangular ones. We can also view the singular values of a (rectangular) matrix as analogues of the eigenvalues of a square matrix. Our definitions and theorems in this section rely heavily on the properties of the matrix-adjoint products ($A^{\ast }A$ and $A{A}^{\ast }$), which we first met in Theorem CPSM. We start by examining some of the basic properties of these two matrices. Now would be a good time to review the basic facts about positive semi-definite matrices in Section PSM.

Subsection MAP: Matrix-Adjoint Product

Theorem EEMAP
Eigenvalues and Eigenvectors of Matrix-Adjoint Product
Suppose that $A$ is an $m\times n$ matrix and $A^{\ast }A$ has rank $r$. Let ${\lambda }_1,{\lambda }_2,{\lambda }_3,\dots ,{\lambda }_p$ be the nonzero distinct eigenvalues of $A^{\ast }A$ and let ${\rho }_1,{\rho }_2,{\rho }_3,\dots ,{\rho }_q$ be the nonzero distinct eigenvalues of $A{A}^{\ast }$. Then,

1. $p=q$.
2. The distinct nonzero eigenvalues can be ordered such that ${\lambda }_i={\rho }_i$, $1\le i\le p$.
3. Properly ordered, ${\alpha }_{A^{\ast }A}\left({\lambda }_i\right)={\alpha }_{A{A}^{\ast }}\left({\rho }_i\right)$, $1\le i\le p$.
4. The rank of $A^{\ast }A$ is equal to the rank of $A{A}^{\ast }$.
5. There is an orthonormal basis, $\left\{x_1,x_2,x_3,\dots ,x_n\right\}$ of ${ℂ}^n$ composed of eigenvectors of $A^{\ast }A$ and an orthonormal basis, $\left\{y_1,y_2,y_3,\dots ,y_m\right\}$ of ${ℂ}^m$ composed of eigenvectors of $A{A}^{\ast }$ with the following properties. Order the eigenvectors so that $x_i$, $r+1\le i\le n$ are the eigenvectors of $A^{\ast }A$ for the zero eigenvalue. Let ${\delta }_i$, $1\le i\le r$ denote the nonzero eigenvalues of $A^{\ast }A$. Then $A{x}_i=\sqrt{{\delta }_i}{y}_i$, $1\le i\le r$ and $A{x}_i=0$, $r+1\le i\le n$. Finally, $y_i$, $r+1\le i\le m$, are eigenvectors of $A{A}^{\ast }$ for the zero eigenvalue.

Proof   Suppose that $x\in {ℂ}^n$ is any eigenvector of $A^{\ast }A$ for a nonzero eigenvalue $\lambda$. We will show that $Ax$ is an eigenvector of $A{A}^{\ast }$ for the same eigenvalue, $\lambda$. First, we ascertain that $Ax$ is not the zero vector.

Since $x$ is an eigenvector, $x\ne 0$, and by Theorem PIP, $〈x,x〉\ne 0$. As $\lambda$ was assumed to be nonzero, we see that $〈Ax,Ax〉\ne 0$. Again, Theorem PIP tells us that $Ax\ne 0$.

Much of the sequel turns on the following simple computation. If you ever wonder what all the fuss is about adjoints, Hermitian matrices, square roots, and singular values, return to this brief computation, as it holds the key. There is much more to do in this proof, but after this it is mostly bookkeeping. Here we go. We check that $Ax$ functions as an eigenvector of $A{A}^{\ast }$ for the eigenvalue $\lambda$,

That’s it. If $x$ is an eigenvector of $A^{\ast }A$ (for a nonzero eigenvalue), then $Ax$ is an eigenvector for $A{A}^{\ast }$ for the same eigenvalue. Let’s see what this buys us.

$A^{\ast }A$ and $A{A}^{\ast }$ are Hermitian matrices (Definition HM), and hence are normal (Definition NRML). This provides the existence of orthonormal bases of eigenvectors for each matrix by Theorem OBNM. Also, since each matrix is diagonalizable (Definition DZM) by Theorem OD we can interchange algebraic and geometric multiplicities by Theorem DMFE.

Our first step is to establish that an eigenvalue $\lambda$ has the same geometric multiplicity for both $A^{\ast }A$ and $A{A}^{\ast }$. Suppose $\left\{x_1,x_2,x_3,\dots ,x_s\right\}$ is an orthonormal basis of eigenvectors of $A^{\ast }A$ for the eigenspace ${\mathcal{ℰ}}_{A^{\ast }A}\left(\lambda \right)$. Then for $1\le i<j\le s$, note

Then the set $E=\left\{A{x}_1,A{x}_2,A{x}_3,\dots ,A{x}_s\right\}$ is an orthogonal set of nonzero eigenvectors of $A{A}^{\ast }$ for the eigenvalue $\lambda$. By Theorem OSLI, the set $E$ is linearly independent and so the geometric multiplicity of $\lambda$ as an eigenvalue of $A{A}^{\ast }$ is $s$ or greater. We have

This inequality applies to any matrix, so long as the eigenvalue is nonzero. We now apply it to the matrix $A^{\ast }$,

So for a nonzero eigenvalue, its algebraic multiplicities as an eigenvalue of $A^{\ast }A$ and $A{A}^{\ast }$ are equal. This is enough to establish that $p=q$ and the eigenvalues can be ordered such that ${\lambda }_i={\rho }_i$ for $1\le i\le p$.

For any matrix $B$, the null space is identical to the eigenspace of the zero eigenvalue, $\mathcal{N}\left(B\right)={\mathcal{ℰ}}_B\left(0\right)$, and thus the nullity of the matrix is equal to the geometric multiplicity of the zero eigenvalue. With this, we can examine the ranks of $A^{\ast }A$ and $A{A}^{\ast }$.

When $A$ is rectangular, the square matrices $A^{\ast }A$ and $A{A}^{\ast }$ have different sizes. With equal algebraic and geometric multiplicities for their common nonzero eigenvalues, the difference in their sizes is manifest in different algebraic multiplicities for the zero eigenvalue and different nullities. Specifically,

Suppose that $x_1,x_2,x_3,\dots ,x_n$ is an orthonormal basis of ${ℂ}^n$ composed of eigenvectors of $A^{\ast }A$ and ordered so that $x_i$, $r+1\le i\le n$ are eigenvectors of $A{A}^{\ast }$ for the zero eigenvalue. Denote the associated nonzero eigenvalues of $A^{\ast }A$ for these eigenvectors by ${\delta }_i$, $1\le i\le r$. Then define

Let $y_{r+1},y_{r+2},y_{r+2},\dots ,y_m$ be an orthonormal basis for the eigenspace ${\mathcal{ℰ}}_{A{A}^{\ast }}\left(0\right)$, whose existence is guaranteed by Theorem GSP. As scalar multiples of demonstrated eigenvectors of $A{A}^{\ast }$, $y_i$, $1\le i\le r$ are also eigenvectors of $A{A}^{\ast }$, and $y_i$, $r+1\le i\le n$ have been chosen as eigenvectors of $A{A}^{\ast }$. These eigenvectors also have norm 1, as we now show. For $1\le i\le r$,

For $r+1\le i\le n$, the $y_i$ have been chosen to have norm 1.

Finally we check orthogonality. Consider two eigenvectors $y_i$ and $y_j$ with $1\le i<j\le m$. If these two vectors have different eigenvalues, then Theorem HMOE establishes that the two eigenvectors are orthogonal. If the two eigenvectors have a zero eigenvalue, then they are orthogonal by the choice of the orthonormal basis of ${\mathcal{ℰ}}_{A{A}^{\ast }}\left(0\right)$. If the two eigenvectors have identical, nonzero, eigenvalues, then

So $\left\{y_1,y_2,y_3,\dots ,y_m\right\}$ is an orthonormal set of eigenvectors for $A{A}^{\ast }$. The critical relationship between these two orthonormal bases is present by design. For $1\le i\le r$,

For $r+1\le i\le n$ we have

So by Theorem PIP, $A{x}_i=0$. $■$

Subsection SVD: Singular Value Decomposition

The square roots of the eigenvalues of $A^{\ast }A$ (or almost equivalently, $A{A}^{\ast }$!) are known as the singular values of $A$. Here is the definition.

Definition SV
Singular Values
Suppose $A$ is an $m\times n$ matrix. If the eigenvalues of $A^{\ast }A$ are ${\delta }_1,{\delta }_2,{\delta }_3,\dots ,{\delta }_n$, then the singular values of $A$ are $\sqrt{{\delta }_1},\sqrt{{\delta }_2},\sqrt{{\delta }_3},\dots ,\sqrt{{\delta }_n}$. $△$

Theorem EEMAP is a total setup for the singular value decomposition. This remarkable theorem says that any matrix can be broken into a product of three matrices. Two are square, and unitary. In light of Theorem UMPIP, we can view these matrices as transforming vectors or coordinates in a rotational fashion. The middle matrix of this decomposition is rectangular, but is as close to being diagonal as a rectangular matrix can be. Viewed as a transformation, this matrix effects, relections, contractions or expansions along axes — it stretches vectors. So any matrix, viewed as a transformation is the product of a rotation, a stretch and a rotation.

The singular value theorem can also be viewed as an application of our most general statement about matrix representations of linear transformations relative to different bases. Theorem MRCB concerns linear transformations $T:U\mapsto V$ where $U$ and $V$ are possibly different vector spaces. When $U$ and $V$ have different dimensions, the resulting matrix representation will be rectangular. In Section CB we quickly specialized to the case where $U=V$ and the matrix representations are square with one of our most central results, Theorem SCB. Theorem SVD is an application of the full generality of Theorem MRCB where the relevant bases are now orthonormal sets.

Theorem SVD
Singular Value Decomposition
Suppose $A$ is an $m\times n$ matrix of rank $r$ with nonzero singular values $s_1,s_2,s_3,\dots ,s_r$. Then $A=UD{V}^{\ast }$ where $U$ is a unitary matrix of size $m$, $V$ is a unitary matrix of size $n$ and $D$ is an $m\times n$ matrix given by

Proof   Let $x_1,x_2,x_3,\dots ,x_n$ and $y_1,y_2,y_3,\dots ,y_m$ be the orthonormal bases described by the conclusion of Theorem EEMAP. Define $U$ to be the $m\times m$ matrix whose columns are $y_i$, $1\le i\le m$, and define $V$ to be the $n\times n$ matrix whose columns are $x_i$, $1\le i\le n$. With orthonormal sets of columns, by Theorem CUMOS both $U$ and $V$ are unitary matrices.

Then for $1\le i\le m$, $1\le j\le n$,

So by Theorem ME, $AV=DU$ and thus