Section FS  Four Subsets

From A First Course in Linear Algebra
Version 1.21
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

There are four natural subsets associated with a matrix. We have met three already: the null space, the column space and the row space. In this section we will introduce a fourth, the left null space. The objective of this section is to describe one procedure that will allow us to find linearly independent sets that span each of these four sets of column vectors. Along the way, we will make a connection with the inverse of a matrix, so Theorem FS will tie together most all of this chapter (and the entire course so far).

Subsection LNS: Left Null Space

Definition LNS
Left Null Space
Suppose $A$ is an $m\times n$ matrix. Then the left null space is defined as $\mathcal{ℒ}\left(A\right)=\mathcal{N}\left(A^t\right)\subseteq {ℂ}^m$.

(This definition contains Notation LNS.) $△$

The left null space will not feature prominently in the sequel, but we can explain its name and connect it to row operations. Suppose $y\in \mathcal{ℒ}\left(A\right)$. Then by Definition LNS, $A^{t}y=0$. We can then write

The product $y^{t}A$ can be viewed as the components of $y$ acting as the scalars in a linear combination of the rows of $A$. And the result is a “row vector”, $0^t$ that is totally zeros. When we apply a sequence of row operations to a matrix, each row of the resulting matrix is some linear combination of the rows. These observations tell us that the vectors in the left null space are scalars that record a sequence of row operations that result in a row of zeros in the row-reduced version of the matrix. We will see this idea more explicitly in the course of proving Theorem FS.

Example LNS
Left null space
We will find the left null space of

We transpose $A$ and row-reduce,

Applying Definition LNS and Theorem BNS we have

If you row-reduce $A$ you will discover one zero row in the reduced row-echelon form. This zero row is created by a sequence of row operations, which in total amounts to a linear combination, with scalars $a_1=-2$, $a_2=3$, $a_3=-1$ and $a_4=1$, on the rows of $A$ and which results in the zero vector (check this!). So the components of the vector describing the left null space of $A$ provide a relation of linear dependence on the rows of $A$. $⊠$

Subsection CRS: Computing Column Spaces

We have three ways to build the column space of a matrix. First, we can use just the definition, Definition CSM, and express the column space as a span of the columns of the matrix. A second approach gives us the column space as the span of some of the columns of the matrix, but this set is linearly independent (Theorem BCS). Finally, we can transpose the matrix, row-reduce the transpose, kick out zero rows, and transpose the remaining rows back into column vectors. Theorem CSRST and Theorem BRS tell us that the resulting vectors are linearly independent and their span is the column space of the original matrix.

We will now demonstrate a fourth method by way of a rather complicated example. Study this example carefully, but realize that its main purpose is to motivate a theorem that simpifies much of the apparent complexity. So other than an instructive exercise or two, the procedure we are about to describe will not be a usual approach to computing a column space.

Example CSANS
Column space as null space
Lets find the column space of the matrix $A$ below with a new approach.

By Theorem CSCS we know that the column vector $b$ is in the column space of $A$ if and only if the linear system $\mathcal{ℒ}\mathcal{S}\left(A,b\right)$ is consistent. So let’s try to solve this system in full generality, using a vector of variables for the vector of constants. In other words, which vectors $b$ lead to consistent systems? Begin by forming the augmented matrix $\left[\left.A\right|b\right]$ with a general version of $b$,

To identify solutions we will row-reduce this matrix and bring it to reduced row-echelon form. Despite the presence of variables in the last column, there is nothing to stop us from doing this. Except our numerical routines on calculators can’t be used, and even some of the symbolic algebra routines do some unexpected maneuvers with this computation. So do it by hand. Yes, it is a bit of work. But worth it. We’ll still be here when you get back. Notice along the way that the row operations are exactly the same ones you would do if you were just row-reducing the coefficient matrix alone, say in connection with a homogeneous system of equations. The column with the $b_i$ acts as a sort of bookkeeping device. There are many different possibilities for the result, depending on what order you choose to perform the row operations, but shortly we’ll all be on the same page. Here’s one possibility (you can find this same result by doing additional row operations with the fifth and sixth rows to remove any occurences of $b_1$ and $b_2$ from the first four rows of your result):

Our goal is to identify those vectors $b$ which make $\mathcal{ℒ}\mathcal{S}\left(A,b\right)$ consistent. By Theorem RCLS we know that the consistent systems are precisely those without a leading 1 in the last column. Are the expressions in the last column of rows 5 and 6 equal to zero, or are they leading 1’s? The answer is: maybe. It depends on $b$. With a nonzero value for either of these expressions, we would scale the row and produce a leading 1. So we get a consistent system, and $b$ is in the column space, if and only if these two expressions are both simultaneously zero. In other words, members of the column space of $A$ are exactly those vectors $b$ that satisfy

Hmmm. Looks suspiciously like a homogeneous system of two equations with six variables. If you’ve been playing along (and we hope you have) then you may have a slightly different system, but you should have just two equations. Form the coefficient matrix and row-reduce (notice that the system above has a coefficient matrix that is already in reduced row-echelon form). We should all be together now with the same matrix,

So, $\mathcal{C}\left(A\right)=\mathcal{N}\left(L\right)$ and we can apply Theorem BNS to obtain a linearly independent set to use in a span construction,

Whew! As a postscript to this central example, you may wish to convince yourself that the four vectors above really are elements of the column space? Do they create consistent systems with $A$ as coefficient matrix? Can you recognize the constant vector in your description of these solution sets?

OK, that was so much fun, let’s do it again. But simpler this time. And we’ll all get the same results all the way through. Doing row operations by hand with variables can be a bit error prone, so let’s see if we can improve the process some. Rather than row-reduce a column vector $b$ full of variables, let’s write $b=I_{6}b$ and we will row-reduce the matrix $I_6$ and when we finish row-reducing, then we will compute the matrix-vector product. You should first convince yourself that we can operate like this (this is the subject of a future homework exercise). Rather than augmenting $A$ with $b$, we will instead augment it with $I_6$ (does this feel familiar?),

We want to row-reduce the left-hand side of this matrix, but we will apply the same row operations to the right-hand side as well. And once we get the left-hand side in reduced row-echelon form, we will continue on to put leading 1’s in the final two rows, as well as clearing out the columns containing those two additional leading 1’s. It is these additional row operations that will ensure that we all get to the same place, since the reduced row-echelon form is unique (Theorem RREFU),

We are after the final six columns of this matrix, which we will multiply by $b$

so

So by applying to the identity matrix the same row operations that row-reduce $A$ (which we could do with a calculator once $I_6$ is placed alongside of $A$), we can then arrive at the result of row-reducing a column of symbols where the vector of constants usually resides. Since the row-reduced version of $A$ has two zero rows, for a consistent system we require that

Now we are exactly back where we were on the first go-round. Notice that we obtain the matrix $L$ as simply the last two rows and last six columns of $N$. $⊠$

This example motivates the remainder of this section, so it is worth careful study. You might attempt to mimic the second approach with the coefficient matrices of Archetype I and Archetype J. We will see shortly that the matrix $L$ contains more information about $A$ than just the column space.

Subsection EEF: Extended echelon form

The final matrix that we row-reduced in Example CSANS should look familiar in most respects to the procedure we used to compute the inverse of a nonsingular matrix, Theorem CINM. We will now generalize that procedure to matrices that are not necessarily nonsingular, or even square. First a definition.

Definition EEF
Extended Echelon Form
Suppose $A$ is an $m\times n$ matrix. Add $m$ new columns to $A$ that together equal an $m\times m$ identity matrix to form an $m\times \left(n+m\right)$ matrix $M$. Use row operations to bring $M$ to reduced row-echelon form and call the result $N$. $N$ is the extended reduced row-echelon form of $A$, and we will standardize on names for five submatrices ($B$, $C$, $J$, $K$, $L$) of $N$.

Let $B$ denote the $m\times n$ matrix formed from the first $n$ columns of $N$ and let $J$ denote the $m\times m$ matrix formed from the last $m$ columns of $N$. Suppose that $B$ has $r$ nonzero rows. Further partition $N$ by letting $C$ denote the $r\times n$ matrix formed from all of the non-zero rows of $B$. Let $K$ be the $r\times m$ matrix formed from the first $r$ rows of $J$, while $L$ will be the $\left(m-r\right)\times m$ matrix formed from the bottom $m-r$ rows of $J$. Pictorially,

Example SEEF
Submatrices of extended echelon form
We illustrate Definition EEF with the matrix $A$,

Augmenting with the $4\times 4$ identity matrix, M=

and row-reducing, we obtain

So we then obtain

You can observe (or verify) the properties of the following theorem with this example. $⊠$

Theorem PEEF
Properties of Extended Echelon Form
Suppose that $A$ is an $m\times n$ matrix and that $N$ is its extended echelon form. Then

1. $J$ is nonsingular.
2. $B=JA$.
3. If $x\in {ℂ}^n$ and $y\in {ℂ}^m$, then $Ax=y$ if and only if $Bx=Jy$.
4. $C$ is in reduced row-echelon form, has no zero rows and has $r$ pivot columns.
5. $L$ is in reduced row-echelon form, has no zero rows and has $m-r$ pivot columns.

Proof   $J$ is the result of applying a sequence of row operations to $I_m$, as such $J$ and $I_m$ are row-equivalent. $\mathcal{ℒ}\mathcal{S}\left(I_m,0\right)$ has only the zero solution, since $I_m$ is nonsingular (Theorem NMRRI). Thus, $\mathcal{ℒ}\mathcal{S}\left(J,0\right)$ also has only the zero solution (Theorem REMES, Definition ESYS) and $J$ is therefore nonsingular (Definition NSM).

To prove the second part of this conclusion, first convince yourself that row operations and the matrix-vector are commutative operations. By this we mean the following. Suppose that $F$ is an $m\times n$ matrix that is row-equivalent to the matrix $G$. Apply to the column vector $Fw$ the same sequence of row operations that converts $F$ to $G$. Then the result is $Gw$. So we can do row operations on the matrix, then do a matrix-vector product, or do a matrix-vector product and then do row operations on a column vector, and the result will be the same either way. Since matrix multiplication is defined by a collection of matrix-vector products (Definition MM), if we apply to the matrix product $FH$ the same sequence of row operations that converts $F$ to $G$ then the result will equal $GH$. Now apply these observations to $A$.

Write $A{I}_n=I_{m}A$ and apply the row operations that convert $M$ to $N$. $A$ is converted to $B$, while $I_m$ is converted to $J$, so we have $B{I}_n=JA$. Simplifying the left side gives the desired conclusion.

For the third conclusion, we now establish the two equivalences

The forward direction of the first equivalence is accomplished by multiplying both sides of the matrix equality by $J$, while the backward direction is accomplished by multiplying by the inverse of $J$ (which we know exists by Theorem NI since $J$ is nonsingular). The second equivalence is obtained simply by the substitutions given by $JA=B$.

The first $r$ rows of $N$ are in reduced row-echelon form, since any contiguous collection of rows taken from a matrix in reduced row-echelon form will form a matrix that is again in reduced row-echelon form. Since the matrix $C$ is formed by removing the last $n$ entries of each these rows, the remainder is still in reduced row-echelon form. By its construction, $C$ has no zero rows. $C$ has $r$ rows and each contains a leading 1, so there are $r$ pivot columns in $C$.

The final $m-r$ rows of $N$ are in reduced row-echelon form, since any contiguous collection of rows taken from a matrix in reduced row-echelon form will form a matrix that is again in reduced row-echelon form. Since the matrix $L$ is formed by removing the first $n$ entries of each these rows, and these entries are all zero (they form the zero rows of $B$), the remainder is still in reduced row-echelon form. $L$ is the final $m-r$ rows of the nonsingular matrix $J$, so none of these rows can be totally zero, or $J$ would not row-reduce to the identity matrix. $L$ has $m-r$ rows and each contains a leading 1, so there are $m-r$ pivot columns in $L$. $■$

Notice that in the case where $A$ is a nonsingular matrix we know that the reduced row-echelon form of $A$ is the identity matrix (Theorem NMRRI), so $B=I_n$. Then the second conclusion above says $JA=B=I_n$, so $J$ is the inverse of $A$. Thus this theorem generalizes Theorem CINM, though the result is a “left-inverse” of $A$ rather than a “right-inverse.”

The third conclusion of Theorem PEEF is the most telling. It says that $x$ is a solution to the linear system $\mathcal{ℒ}\mathcal{S}\left(A,y\right)$ if and only if $x$ is a solution to the linear system $\mathcal{ℒ}\mathcal{S}\left(B,Jy\right)$. Or said differently, if we row-reduce the augmented matrix $\left[\left.A\right|x\right]$ we will get the augmented matrix $\left[\left.B\right|Jy\right]$. The matrix $J$ tracks the cumulative effect of the row operations that converts $A$ to reduced row-echelon form, here effectively applying them to the vector of constants in a system of equations having $A$ as a coefficient matrix. When $A$ row-reduces to a matrix with zero rows, then $Jy$ should also have zero entries in the same rows if the system is to be consistent.

Subsection FS: Four Subsets

With all the preliminaries in place we can state our main result for this section. In essence this result will allow us to say that we can find linearly independent sets to use in span constructions for all four subsets (null space, column space, row space, left null space) by analyzing only the extended echelon form of the matrix, and specifically, just the two submatrices $C$ and $L$, which will be ripe for analysis since they are already in reduced row-echelon form (Theorem PEEF).

Theorem FS
Four Subsets
Suppose $A$ is an $m\times n$ matrix with extended echelon form $N$. Suppose the reduced row-echelon form of $A$ has $r$ nonzero rows. Then $C$ is the submatrix of $N$ formed from the first $r$ rows and the first $n$ columns and $L$ is the submatrix of $N$ formed from the last $m$ columns and the last $m-r$ rows. Then

1. The null space of $A$ is the null space of $C$, $\mathcal{N}\left(A\right)=\mathcal{N}\left(C\right)$.
2. The row space of $A$ is the row space of $C$, $\mathcal{ℛ}\left(A\right)=\mathcal{ℛ}\left(C\right)$.
3. The column space of $A$ is the null space of $L$, $\mathcal{C}\left(A\right)=\mathcal{N}\left(L\right)$.
4. The left null space of $A$ is the row space of $L$, $\mathcal{ℒ}\left(A\right)=\mathcal{ℛ}\left(L\right)$.

Proof   First, $\mathcal{N}\left(A\right)=\mathcal{N}\left(B\right)$ since $B$ is row-equivalent to $A$ (Theorem REMES). The zero rows of $B$ represent equations that are always true in the homogeneous system $\mathcal{ℒ}\mathcal{S}\left(B,0\right)$, so the removal of these equations will not change the solution set. Thus, in turn, $\mathcal{N}\left(B\right)=\mathcal{N}\left(C\right)$.

Second, $\mathcal{ℛ}\left(A\right)=\mathcal{ℛ}\left(B\right)$ since $B$ is row-equivalent to $A$ (Theorem REMRS). The zero rows of $B$ contribute nothing to the span that is the row space of $B$, so the removal of these rows will not change the row space. Thus, in turn, $\mathcal{ℛ}\left(B\right)=\mathcal{ℛ}\left(C\right)$.

Third, we prove the set equality $\mathcal{C}\left(A\right)=\mathcal{N}\left(L\right)$ with Definition SE. Begin by showing that $\mathcal{C}\left(A\right)\subseteq \mathcal{N}\left(L\right)$. Choose $y\in \mathcal{C}\left(A\right)\subseteq {ℂ}^m$. Then there exists a vector $x\in {ℂ}^n$ such that $Ax=y$ (Theorem CSCS). Then for $1\le k\le m-r$,

So, for all $1\le k\le m-r$, ${\left[Ly\right]}_k={\left[0\right]}_k$. So by Definition CVE we have $Ly=0$ and thus $y\in \mathcal{N}\left(L\right)$.

Now, show that $\mathcal{N}\left(L\right)\subseteq \mathcal{C}\left(A\right)$. Choose $y\in \mathcal{N}\left(L\right)\subseteq {ℂ}^m$. Form the vector $Ky\in {ℂ}^r$. The linear system $\mathcal{ℒ}\mathcal{S}\left(C,Ky\right)$ is consistent since $C$ is in reduced row-echelon form and has no zero rows (Theorem PEEF). Let $x\in {ℂ}^n$ denote a solution to $\mathcal{ℒ}\mathcal{S}\left(C,Ky\right)$.

Then for $1\le j\le r$,

And for $r+1\le k\le m$,

So for all $1\le i\le m$, ${\left[Bx\right]}_i={\left[Jy\right]}_i$ and by Definition CVE we have $Bx=Jy$. From Theorem PEEF we know then that $Ax=y$, and therefore $y\in \mathcal{C}\left(A\right)$ (Theorem CSCS). By Definition SE we now have $\mathcal{C}\left(A\right)=\mathcal{N}\left(L\right)$.

Fourth, we prove the set equality $\mathcal{ℒ}\left(A\right)=\mathcal{ℛ}\left(L\right)$ with Definition SE. Begin by showing that $\mathcal{ℛ}\left(L\right)\subseteq \mathcal{ℒ}\left(A\right)$. Choose $y\in \mathcal{ℛ}\left(L\right)\subseteq {ℂ}^m$. Then there exists a vector $w\in {ℂ}^{m-r}$ such that $y=L^{t}w$ (Definition RSM, Theorem CSCS). Then for $1\le i\le n$,

Since ${\left[A^{t}y\right]}_i={\left[0\right]}_i$ for $1\le i\le n$, Definition CVE implies that $A^{t}y=0$. This means that $y\in \mathcal{N}\left(A^t\right)$.

Now, show that $\mathcal{ℒ}\left(A\right)\subseteq \mathcal{ℛ}\left(L\right)$. Choose $y\in \mathcal{ℒ}\left(A\right)\subseteq {ℂ}^m$. The matrix $J$ is nonsingular (Theorem PEEF), so $J^t$ is also nonsingular (Theorem MIT) and therefore the linear system $\mathcal{ℒ}\mathcal{S}\left(J^t,y\right)$ has a unique solution. Denote this solution as $x\in {ℂ}^m$. We will need to work with two “halves” of $x$, which we will denote as $z$ and $w$ with formal definitions given by

Now, for $1\le j\le r$,

So, by Definition CVE, $C^{t}z=0$ and the vector $z$ gives us a linear combination of the columns of $C^t$ that equals the zero vector. In other words, $z$ gives a relation of linear dependence on the the rows of $C$. However, the rows of $C$ are a linearly independent set by Theorem BRS. According to Definition LICV we must conclude that the entries of $z$ are all zero, i.e. $z=0$.

Now, for $1\le i\le m$, we have

So by Definition CVE, $y=L^{t}w$. The existence of $w$ implies that $y\in \mathcal{ℛ}\left(L\right)$, and therefore $\mathcal{ℒ}\left(A\right)\subseteq \mathcal{ℛ}\left(L\right)$. So by Definition SE we have $\mathcal{ℒ}\left(A\right)=\mathcal{ℛ}\left(L\right)$. $■$

The first two conclusions of this theorem are nearly trivial. But they set up a pattern of results for $C$ that is reflected in the latter two conclusions about $L$. In total, they tell us that we can compute all four subsets just by finding null spaces and row spaces. This theorem does not tell us exactly how to compute these subsets, but instead simply expresses them as null spaces and row spaces of matrices in reduced row-echelon form without any zero rows ($C$ and $L$). A linearly independent set that spans the null space of a matrix in reduced row-echelon form can be found easily with Theorem BNS. It is an even easier matter to find a linearly independent set that spans the row space of a matrix in reduced row-echelon form with Theorem BRS, especially when there are no zero rows present. So an application of Theorem FS is typically followed by two applications each of Theorem BNS and Theorem BRS.

The situation when $r=m$ deserves comment, since now the matrix $L$ has no rows. What is $\mathcal{C}\left(A\right)$ when we try to apply Theorem FS and encounter $\mathcal{N}\left(L\right)$? One interpretation of this situation is that $L$ is the coefficient matrix of a homogeneous system that has no equations. How hard is it to find a solution vector to this system? Some thought will convince you that any proposed vector will qualify as a solution, since it makes all of the equations true. So every possible vector is in the null space of $L$ and therefore $\mathcal{C}\left(A\right)=\mathcal{N}\left(L\right)={ℂ}^m$. OK, perhaps this sounds like some twisted argument from Alice in Wonderland. Let us try another argument that might solidly convince you of this logic.

If $r=m$, when we row-reduce the augmented matrix of $\mathcal{ℒ}\mathcal{S}\left(A,b\right)$ the result will have no zero rows, and all the leading 1’s will occur in first $n$ columns, so by Theorem RCLS the system will be consistent. By Theorem CSCS, $b\in \mathcal{C}\left(A\right)$. Since $b$ was arbitrary, every possible vector is in the column space of $A$, so we again have $\mathcal{C}\left(A\right)={ℂ}^m$. The situation when a matrix has $r=m$ is known by the term full rank, and in the case of a square matrix coincides with nonsingularity (see Exercise FS.M50).

The properties of the matrix $L$ described by this theorem can be explained informally as follows. A column vector $y\in {ℂ}^m$ is in the column space of $A$ if the linear system $\mathcal{ℒ}\mathcal{S}\left(A,y\right)$ is consistent (Theorem CSCS). By Theorem RCLS, the reduced row-echelon form of the augmented matrix $\left[\left.A\right|y\right]$ of a consistent system will have zeros in the bottom $m-r$ locations of the last column. By Theorem PEEF this final column is the vector $Jy$ and so should then have zeros in the final $m-r$ locations. But since $L$ comprises the final $m-r$ rows of $J$, this condition is expressed by saying $y\in \mathcal{N}\left(L\right)$.

Additionally, the rows of $J$ are the scalars in linear combinations of the rows of $A$ that create the rows of $B$. That is, the rows of $J$ record the net effect of the sequence of row operations that takes $A$ to its reduced row-echelon form, $B$. This can be seen in the equation $JA=B$ (Theorem PEEF). As such, the rows of $L$ are scalars for linear combinations of the rows of $A$ that yield zero rows. But such linear combinations are precisely the elements of the left null space. So any element of the row space of $L$ is also an element of the left null space of $A$. We will now illustrate Theorem FS with a few examples.

Example FS1
Four subsets, #1
In Example SEEF we found the five relevant submatrices of the matrix

To apply Theorem FS we only need $C$ and $L$,

Then we use Theorem FS to obtain

Boom! $⊠$

Example FS2
Four subsets, #2
Now lets return to the matrix $A$ that we used to motivate this section in Example CSANS,

We form the matrix $M$ by adjoining the $6\times 6$ identity matrix $I_6$,

and row-reduce to obtain $N$

To find the four subsets for $A$, we only need identify the $4\times 5$ matrix $C$ and the $2\times 6$ matrix $L$,

Then we apply Theorem FS,