Section TD  Triangular Decomposition

From A First Course in Linear Algebra
Version 2.00
© 2004.
Licensed under the GNU Free Documentation License.

This Section is a Draft, Subject to Changes

Our next decomposition will break a square matrix into a product of two matrices, one lower triangular and the other upper triangular. So we will write A = LU, and hence many refer to this as LU decomposition. We will see that this decomposition is very easy to compute and that it has a direct application to solving systems of equations. Since this section is about triangular matrices you might want to review the definitions and a couple of basic theorems back in Subsection OD.TM.

Subsection TD: Triangular Decomposition

With a slight condition on the nonsingularity of certain submatrices, we can split a matrix into a product of two triangular matrices.

Theorem TD
Triangular Decomposition
Suppose A is a square matrix of size n. Let Ak be the k × k matrix formed from A by taking the first k rows and the first k columns. Suppose that Ak is nonsingular for all 1 k n. Then there is a lower triangular matrix L with all of its diagonal entries equal to 1 and an upper triangular matrix U such that A = LU. Furthermore, this decomposition is unique.

Proof   We will row reduce A to a row-equivalent upper triangular matrix through a series of row operations, forming intermediate matrices Aj, 1 j n, that denote the state of the conversion after working on column j. First, the lone entry of A1 is A11 and this scalar must be nonzero if A1 is nonsingular (Theorem SMZD). We can use row operations Definition RO of the form αR1 + Rk, 2 k n, where α = A1k A11 to place zeros in the first column below the diagonal. The first two rows and columns of A1 are a 2 × 2 upper triangular matrix whose determinant is equal to the determinant of A2, since the matrices are row-equivalent through a sequence of row operations strictly of the third type (Theorem DRCMA). As such the diagonal entries of this 2 × 2 submatrix of A1 are nonzero. We can employ this nonzero diagonal element with row operations of the form αR2 + Rk, 3 k n to place zeros below the diagonal in the second column. We can continue this process, column by column. The key observations are that our hypothesis on the nonsingularity of the Ak will guarantee a nonzero diagonal entry for each column when we need it, that the row operations employed are always of the third type using a multiple of a row to transform another row with a greater row index, and that the final result will be a nonsingular upper triangular matrix. This is the desired matrix U.

Each row operation described in the previous paragraph can be accompished with matrix multiplication by the appropriate elementary matrix (Theorem EMDRO). Since every row operation employed is adding a multiple of a row to a subsequent row these elementary matrices are of the form Ej,k α with j < k. By Definition ELEM, these matrices are lower triangular with every diagonal entry equal to 1. We know that the product of two such matrices will again be lower triangular (Theorem PTMT), but also, as you can also easily check using a proof with a style similar to one above, that the product maintains all 1’s on the diagonal. Let E1,E2,E3,,Em denote the elementary matrices for this sequence of row operations. Then

U = EmEm1E3E2E1A = LA

where L is the product of the elementary matrices, and we know L is lower triangular with all 1’s on the diagonal. Our desired matrix L is then L = L1. By Theorem ITMT, L is lower triangular with all 1’s on the diagonal and A = LU, as desired.

The process just described is deterministic. That is, the proof is constructive, with no freedom for each of us to walk through it differently. But could there be other matrices with the same properties as L and U that give such a decomposition of A. In other words, is the decomposition unique (Technique U)? Suppose that we have two triangular decompositions, A = L1U1 and A = L2U2. Since A is nonsingular, two applications of Theorem NPNT imply that L1,L2,U1,U2 are all nonsingular. We have

L21L 1 = L21I nL1  Theorem MMIM = L21AA1L 1  Definition MI = L21L 2U2 L1U1 1L 1 = L21L 2U2U11L 11L 1  Theorem SS = InU2U11I n  Definition MI = U2U11  Theorem MMIM

Theorem ITMT tells us that L21 is lower triangular and has 1’s as the diagonal entries. By Theorem PTMT, the product L21L 1 is again lower triangular, and it is simple to check (as before) that the diagonal entries of the product are again all 1’s. By the entirely similar process we can conclude that the product U2U11 is upper triangular. Because these two products are equal, their common value is a matrix that is both lower triangular and upper triangular, with all 1’s on the diagonal. The only matrix meeting these three requirements is the identity matrix (Definition IM). So, we have,

In = L21L 1 L2 = L1 In = U2U11 U 1 = U2

which establishes the uniqueness of the decomposition.

Studying the proofs of some previous theorems will perhaps give you an idea for an approach to computing a triangular decomposition. In the proof of Theorem CINM we augmented a nonsingular matrix with an identity matrix of the same size, and row-reduced until the original matrix became the identity matrix (as we knew in advance would happen, since we knew Theorem NMRRI). Theorem PEEF tells us about properties of extended echelon form, and in particular, that B = JA, where A is the matrix that begins on the left, and B is the reduced row-echelon form of A. The matrix J is the result on the right side of the augmented matrix, which is the result of applying the same row operations to the identity matrix. We should recognize now that J is just the product of the elementary matrices (Subsection DM.EM) that perform these row operations. Theorem ITMT used the extended echelon form to discern properties of the inverse of a triangular matrix. Theorem TD proves the existence of a triangular decomposition by applying specific row operations, and tracking the relevant elementary row operations. It is not a great leap to combine these obervations into a computational procedure.

To find the triangular decomposition of A, augment A with the identity matrix of the same size and call this new 2n × n matrix, M. Perform row operations on M that convert the first n columns to an upper triangular matrix. Do this using only row operations that add a scalar multiple of one row to another row with higher index (i.e. lower down). In this way, the last n columns of M will be converted into a lower triangular matrix with 1’s on the diagonal (since M has 1’s in these locations initially). We could think of this process as doing about half of the work required to compute the inverse of A. Take the first n columns of the row-equivalent version of M and call this matrix U. Take the final n columns of the row-equivalent version of M and call this matrix L. Then by a proof employing elementary matrices, or a proof similar in spirit to the one used to prove Theorem PEEF, we arrive at a result similar to the second assertion of Theorem PEEF. Namely, U = LA. Multiplication on the left, by the inverse of L, will give us a decomposition of A (which we know to be unique). Ready? Lets try it.

Example TD4
Triangular decomposition, size 4
In this example, we will illustrate the process for computing a triangular decomposition, as described in the previous paragraphs. Consider the nonsingular square matrix A of size 4,

A = 2 6 8 7 4 161415 6 222326 6 261817

We form M by augmenting A with the size 4 identity matrix I4. We will perform the allowed operations, column by column, only reporting intermediate results as we finish converting each column. It is easy to determine exactly which row operations we perform, since the final four columns contain a record of each such operation. We will not verify our hypotheses about the nonsingularity of the Ak, since if we do not have these conditions, we will reach a stage where a diagonal entry is zero and we cannot create the row operations we need to zero out the bottom portion of the associated column. In other words, we can boldly proceed and the necessity of our hypotheses will become apparent.

M = 2 6 8 7 1000 4 1614150100 6 2223260010 6 2618170001 268 7 1 000 0 4 2 1 2100 0 4 1 5 3010 0 8 6 43001 268 7 1 0 00 0 4 2 1 2 1 00 0 01 4 1110 0 0 2 6 1 201 2687 1 0 00 0 4 2 12 1 00 0 0141110 0 0 0 21421

So at this point, we have U and L,

U = 2687 0 4 2 1 0 014 0 0 0 2 L = 1 0 00 2 1 00 1110 1421

Then by whatever procedure we like (such as Theorem CINM), we find

L = L1 = 10 0 0 2 1 0 0 3 1 1 0 3 221

It is instructive to verify that indeed LU = A.

Subsection TDSSE: Triangular Decomposition and Solving Systems of Equations

In this section we give an explanation of why you might be interested in a triangular decomposition for a matrix. Many of the computational problems in linear algebra revolve around solving large systems of equations, or nearly equivalently, finding inverses of large matrices. Suppose we have a system of equations with coefficient matrix A and vector of constants b, and suppose further that A has the triangular decomposition A = LU.

Let y be the solution to the linear system SL,b, so that by Theorem SLEMM, we have Ly = b. Notice that since L is nonsingular, this solution is unique, and the form of L makes it trivial to solve the system. The first component of y is determined easily, and we can continue on through determining the components of y, without even ever dividing. Now, with y in hand, consider the linear system, SU,y. Let x be the unique solution to this system, so by Theorem SLEMM we have Ux = y. Notice that a system of equations with U as a coefficient matrix is also straightforward to solve, though we will compute the bottom entries of x first, and we will need to divide. The upshot of all this is that x is a solution to SA,b, as we now show,

Ax = LUx = L Ux = Ly = b

An application of Theorem SLEMM demonstrates that x is a solution to SA,b.

Example TDSSE
Triangular decomposition solves a system of equations
Here we illustrate the previous discussion, recycling the decomposition found previously in Example TD4. Consider the linear system SA,b with

A = 2 6 8 7 4 161415 6 222326 6 261817 b = 10 2 1 8

First we solve the system SL,b (see Example TD4 for L),

y1 = 10 2y1 + y2 = 2 3y1 + y2 + y3 = 1 3y1 + 2y2 2y3 + y4 = 8


y1 = 10 y2 = 2 2y1 = 2 2(10) = 18 y3 = 1 3y1 y2 = 1 3(10) 18 = 11 y4 = 8 3y1 2y2 + 2y3 = 8 3(10) 2(18) + 2(11) = 8


y = 10 18 11 8

Then we solve the system SU,y (see Example TD4 for U),

2x1 + 6x2 8x3 + 7x4 = 10 4x2 + 2x3 + x4 = 18 x3 + 4x4 = 11 2x4 = 8


x4 = 82 = 4 x3 = 11 4x4 (1) = 11 4(4) (1) = 5 x2 = 18 2x3 x4 4 = 18 2(5) 4 4 = 1 x1 = 10 6x2 + 8x3 7x4 (2) = 10 6(1) + 8(5) 7(4) (2) = 2

And so

x = 4 5 1 2

is the solution to SU,y and consequently is the unique solution to SA,b, as you can easily verify.

Subsection CTD: Computing Triangular Decompositions

It would be a simple matter to adjust the algorithm for converting a matrix to reduced row-echelon form and obtain an algorithm to compute the triangular decomposition of the matrix, along the lines of Example TD4 and the discussion preceding this example. However, it is possible to obtain relatively simple formulas for the entries of the decomposition, and if computed in the proper order, an implemtation will be straightfoward. We will state the result as a theorem and then give an example of its use.

Theorem TDEE
Triangular Decomposition, Entry by Entry
Suppose that A is a squarematrix of size n with a triangular decomposition A = LU, where L is lower triangular with diagonal entries all equal to 1, and U is upper triangular. Then

Uij = Aij k=1i1 L ik Ukj 1 i j n Lij = 1 Ujj Aij k=1j1 L ik Ukj 1 j < i n

Proof   Consider a single scalar product of an entry of L with an entry of U of the form Lik Ukj. By Definition LTM, if k > i then Lik = 0, while Definition UTM, says that if k > j then Ukj = 0. So we can combine these two facts to assert that if k > min(i,j), Lik Ukj = 0 since at least one term of the product will be zero. Employing this observation,

Aij = k=1n L ik Ukj  Theorem EMP = k=1min(i,j) L ik Ukj

Now, assume that 1 i j n,

Uij = Aij Aij + Uij = Aij k=1min(i,j) L ik Ukj + Uij = Aij k=1i L ik Ukj + Uij = Aij k=1i1 L ik Ukj Lii Uij + Uij = Aij k=1i1 L ik Ukj Uij + Uij = Aij k=1i1 L ik Ukj

And for 1 j < i n,

Lij = 1 Ujj Lij Ujj = 1 Ujj Aij Aij + Lij Ujj = 1 Ujj Aij k=1min(i,j) L ik Ukj + Lij Ujj = 1 Ujj Aij k=1j L ik Ukj + Lij Ujj = 1 Ujj Aij k=1j1 L ik Ukj Lij Ujj + Lij Ujj = 1 Ujj Aij k=1j1 L ik Ukj

At first glance, these formulas may look exceedingly complex. Upon closer examination, it looks even worse. We have expressions for entries of U that depend on other entries of U and also on entries of L. But then the formula for entries of L depend on entries from L and entries from U. Do these formula have circular dependencies? Or perhaps equivalently, how do we get started? The key is to be organized about the computations and employ these two (similar) formulas in a specific order. First compute the first row of L, followed by the first column of U. Then the second row of L, followed by the second column of U. And so on. In this way, all of the values required for each new entry will have already been computed previously.

Of course, the formula for entries of L require division by diagonal entries of U. These entries might be zero, but in this case A is nonsingular and does not have a triangular decomposition. So we need not check the hypothesis carefully and can launch into the arithmetic dictated by the formulas, confident that we will be reminded when a decomposition is not possible. Note that these formula give us all of the values that we need for the decomposition, since we require that L has 1’s on the diagonal. If we replace the 1’s on the diagonal of L by zeros, and add the matrix U, we get an n × n matrix containing all the information we need to ressurect the triangular decomposition. This is mostly a notational convenience, but it is a frequent way of presenting the information. We’ll employ it in the next example.

Example TDEE6
Triangular decomposition, entry by entry, size 6
We illustrate the application of the formulas in Theorem TDEE for the 6 × 6 matrix A.

A = 3 3 3 2 1 0 6 4 5 2 4 2 9 9 7 7 0 1 610 8 10 1 7 6 4 9 210 1 9 3 123212

Using the notational convenience of packaging the two triangular matrices into one matrix, and using the ordering of the computations mentioned above, we display the results after computing a single row and column of each of the two triangular matrices.

3 33210 2 3 2 2 3 3 3 3210 2 2 12 2 2 3 0 22 2 1 3 3 3 3 3210 2 2 12 2 2 3 0 2 1 3 1 22 0 2 12 3 33 3 3 321 0 2 2 12 2 2 3 0 2 1 3 1 22 0 2 1 3 2 121 3 333 3 3 321 0 2 2 12 2 2 3 0 2 1 3 1 22 0 2 1 3 2 121 1 2 3 333 0 3 3 321 0 2 2 12 2 2 3 0 2 1 3 1 22 0 2 1 3 2 121 1 2 3 333 0 2

Splitting out the pieces of this matrix, we have the decomposition,

L = 1 0 0 0 00 2 1 0 0 00 3 0 1 0 00 22 0 1 00 2 12110 3 33301 U = 33321 0 0 212 2 2 0 0 2 1 3 1 0 0 0 2 1 3 0 0 0 0 1 2 0 0 0 0 0 2

The hypotheses of Theorem TD can be weakened slightly to include matrices where not every Ak is nonsingular. The introduces a rearrangement of the rows and columns of A to force as many as possible of the smaller submatrices to be nonsingular. Then permutation matrices also enter into the decomposition. We will not present the details here, but instead suggest consulting a more advanced text on matrix analysis.