Section O  Orthogonality

From A First Course in Linear Algebra
Version 1.21
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

In this section we define a couple more operations with vectors, and prove a few theorems. At first blush these definitions and results will not appear central to what follows, but we will make use of them at key points in the remainder of the course (such as Section MINM, Section OD). Because we have chosen to use $ℂ$ as our set of scalars, this subsection is a bit more, uh, … complex than it would be for the real numbers. We’ll explain as we go along how things get easier for the real numbers $ℝ$. If you haven’t already, now would be a good time to review some of the basic properties of arithmetic with complex numbers described in Section CNO. With that done, we can extend the basics of complex number arithmetic to our study of vectors in ${ℂ}^m$.

Subsection CAV: Complex Arithmetic and Vectors

We know how the addition and multiplication of complex numbers is employed in defining the operations for vectors in ${ℂ}^m$ (Definition CVA and Definition CVSM). We can also extend the idea of the conjugate to vectors.

Definition CCCV
Complex Conjugate of a Column Vector
Suppose that $u$ is a vector from ${ℂ}^m$. Then the conjugate of the vector, $\overline{u}$, is defined by

(This definition contains Notation CCCV.) $△$

With this definition we can show that the conjugate of a column vector behaves as we would expect with regard to vector addition and scalar multiplication.

Theorem CRVA
Conjugation Respects Vector Addition
Suppose $x$ and $y$ are two vectors from ${ℂ}^m$. Then

Proof   For each $1\le i\le m$,

Then by Definition CVE we have $\overline{x+y}=\overline{x}+\overline{y}$. $■$

Theorem CRSM
Conjugation Respects Vector Scalar Multiplication
Suppose $x$ is a vector from ${ℂ}^m$, and $\alpha \in ℂ$ is a scalar. Then

Proof   For $1\le i\le m$,

Then by Definition CVE we have $\overline{\alpha x}=\overline{\alpha }\overline{x}$. $■$

These two theorems together tell us how we can “push” complex conjugation through linear combinations.

Subsection IP: Inner products

Definition IP
Inner Product
Given the vectors $u,v\in {ℂ}^m$ the inner product of $u$ and $v$ is the scalar quantity in ${ℂ}^{}$,

(This definition contains Notation IP.) $△$

This operation is a bit different in that we begin with two vectors but produce a scalar. Computing one is straightforward.

Example CSIP
Computing some inner products
The scalar product of

is

The scalar product of

is

In the case where the entries of our vectors are all real numbers (as in the second part of Example CSIP), the computation of the inner product may look familiar and be known to you as a dot product or scalar product. So you can view the inner product as a generalization of the scalar product to vectors from ${ℂ}^m$ (rather than ${ℝ}^m$).

Also, note that we have chosen to conjugate the entries of the second vector listed in the inner product, while many authors choose to conjugate entries from the first component. It really makes no difference which choice is made, it just requires that subsequent definitions and theorems are consistent with the choice. You can study the conclusion of Theorem IPAC as an explanation of the magnitude of the difference that results from this choice. But be careful as you read other treatments of the inner product or its use in applications, and be sure you know ahead of time which choice has been made.

There are several quick theorems we can now prove, and they will each be useful later.

Theorem IPVA
Inner Product and Vector Addition
Suppose $u,v,w\in {ℂ}^m$. Then

Proof   The proofs of the two parts are very similar, with the second one requiring just a bit more effort due to the conjugation that occurs. We will prove part 2 and you can prove part 1 (Exercise O.T10).

Theorem IPSM
Inner Product and Scalar Multiplication
Suppose $u,v\in {ℂ}^m$ and $\alpha \in {ℂ}^{}$. Then

Proof   The proofs of the two parts are very similar, with the second one requiring just a bit more effort due to the conjugation that occurs. We will prove part 2 and you can prove part 1 (Exercise O.T11).

Theorem IPAC
Inner Product is Anti-Commutative
Suppose that $u$ and $v$ are vectors in ${ℂ}^m$. Then $〈u,v〉=\overline{〈v,u〉}$. $\square$

Proof  

Subsection N: Norm

If treating linear algebra in a more geometric fashion, the length of a vector occurs naturally, and is what you would expect from its name. With complex numbers, we will define a similar function. Recall that if $c$ is a complex number, then $\left|c\right|$ denotes its modulus (Definition MCN).

Definition NV
Norm of a Vector
The norm of the vector $u$ is the scalar quantity in ${ℂ}^{}$

(This definition contains Notation NV.) $△$

Computing a norm is also easy to do.

Example CNSV
Computing the norm of some vectors
The norm of

is

The norm of

is

Notice how the norm of a vector with real number entries is just the length of the vector. Inner products and norms are related by the following theorem.

Theorem IPN
Inner Products and Norms
Suppose that $u$ is a vector in ${ℂ}^m$. Then $\parallel u{\parallel }^2=〈u,u〉$. $\square$

Proof  

When our vectors have entries only from the real numbers Theorem IPN says that the dot product of a vector with itself is equal to the length of the vector squared.

Theorem PIP
Positive Inner Products
Suppose that $u$ is a vector in ${ℂ}^m$. Then $〈u,u〉\ge 0$ with equality if and only if $u=0$. $\square$

Proof   From the proof of Theorem IPN we see that

Since each modulus is squared, every term is positive, and the sum must also be positive. (Notice that in general the inner product is a complex number and cannot be compared with zero, but in the special case of $〈u,u〉$ the result is a real number.) The phrase, “with equality if and only if” means that we want to show that the statement $〈u,u〉=0$ (i.e. with equality) is equivalent (“if and only if”) to the statement $u=0$.

If $u=0$, then it is a straightforward computation to see that $〈u,u〉=0$. In the other direction, assume that $〈u,u〉=0$. As before, $〈u,u〉$ is a sum of moduli. So we have

Now we have a sum of squares equaling zero, so each term must be zero. Then by similar logic, $\left|{\left[u\right]}_i\right|=0$ will imply that ${\left[u\right]}_i=0$, since $0+0i$ is the only complex number with zero modulus. Thus every entry of $u$ is zero and so $u=0$, as desired. $■$

Notice that Theorem PIP contains three implications:

The results contained in Theorem PIP are summarized by saying “the inner product is positive definite.”

Subsection OV: Orthogonal Vectors

“Orthogonal” is a generalization of “perpendicular.” You may have used mutually perpendicular vectors in a physics class, or you may recall from a calculus class that perpendicular vectors have a zero dot product. We will now extend these ideas into the realm of higher dimensions and complex scalars.

Definition OV
Orthogonal Vectors
A pair of vectors, $u$ and $v$, from ${ℂ}^m$ are orthogonal if their inner product is zero, that is, $〈u,v〉=0$. $△$

Example TOV
Two orthogonal vectors
The vectors

are orthogonal since

We extend this definition to whole sets by requiring vectors to be pairwise orthogonal. Despite using the same word, careful thought about what objects you are using will eliminate any source of confusion.

Definition OSV
Orthogonal Set of Vectors
Suppose that $S=\left\{u_1,u_2,u_3,\dots ,u_n\right\}$ is a set of vectors from ${ℂ}^m$. Then $S$ is an orthogonal set if every pair of different vectors from $S$ is orthogonal, that is $〈u_i,u_j〉=0$ whenever $i\ne j$. $△$

We now define the prototypical orthogonal set, which we will reference repeatedly.

Definition SUV
Standard Unit Vectors
Let $e_j\in {ℂ}^m$, $1\le j\le m$ denote the column vectors defined by

Then the set

is the set of standard unit vectors in ${ℂ}^m$. $△$

Notice that $e_j$ is identical to column $j$ of the $m\times m$ identity matrix $I_m$ (Definition IM). This observation will often be useful. It is not hard to see that the set of standard unit vectors is an orthogonal set.

Example SUVOS
Standard Unit Vectors are an Orthogonal Set
Compute the inner product of two distinct vectors from the set of standard unit vectors (Definition SUV), say $e_i$, $e_j$, where $i\ne j$,

So the set $\left\{e_1,e_2,e_3,\dots ,e_m\right\}$ is an orthogonal set. $⊠$

Example AOS
An orthogonal set
The set

is an orthogonal set. Since the inner product is anti-commutative (Theorem IPAC) we can test pairs of different vectors in any order. If the result is zero, then it will also be zero if the inner product is computed in the opposite order. This means there are six pairs of different vectors to use in an inner product computation. We’ll do two and you can practice your inner products on the other four.

So far, this section has seen lots of definitions, and lots of theorems establishing un-surprising consequences of those definitions. But here is our first theorem that suggests that inner products and orthogonal vectors have some utility. It is also one of our first illustrations of how to arrive at linear independence as the conclusion of a theorem.

Theorem OSLI
Orthogonal Sets are Linearly Independent
Suppose that $S=\left\{u_1,u_2,u_3,\dots ,u_n\right\}$ is an orthogonal set of nonzero vectors. Then $S$ is linearly independent. $\square$

Proof   To prove linear independence of a set of vectors, we can appeal to the definition (Definition LICV) and begin with an arbitrary relation of linear dependence (Definition RLDCV),

Then, for every $1\le i\le n$, we have

So we conclude that ${\alpha }_i=0$ for all $1\le i\le n$ in any relation of linear dependence on $S$. But this says that $S$ is a linearly independent set since the only way to form a relation of linear dependence is the trivial way (Definition LICV). Boom! $■$

Subsection GSP: Gram-Schmidt Procedure

The Gram-Schmidt Procedure is really a theorem. It says that if we begin with a linearly independent set of $p$ vectors, $S$, then we can do a number of calculations with these vectors and produce an orthogonal set of $p$ vectors, $T$, so that $〈S〉=〈T〉$. Given the large number of computations involved, it is indeed a procedure to do all the necessary computations, and it is best employed on a computer. However, it also has value in proofs where we may on occasion wish to replace a linearly independent set by an orthogonal set.

This is our first occasion to use the technique of “mathematical induction” for a proof, a technique we will see again several times, especially in Chapter D. So study the simple example described in Technique I first.

Theorem GSP
Gram-Schmidt Procedure
Suppose that $S=\left\{v_1,v_2,v_3,\dots ,v_p\right\}$ is a linearly independent set of vectors in ${ℂ}^m$. Define the vectors $u_i$, $1\le i\le p$ by

Then if $T=\left\{u_1,u_2,u_3,\dots ,u_p\right\}$, then $T$ is an orthogonal set of non-zero vectors, and $〈T〉=〈S〉$. $\square$

Proof   We will prove the result by using induction on $p$ (Technique I). To begin, we prove that $T$ has the desired properties when $p=1$. In this case $u_1=v_1$ and $T=\left\{u_1\right\}=\left\{v_1\right\}=S$. Because $S$ and $T$ are equal, $〈S〉=〈T〉$. Equally trivial, $T$ is an orthogonal set. If $u_1=0$, then $S$ would be a linearly dependent set, a contradiction.

Now suppose that the theorem is true for any set of $p-1$ linearly independent vectors. Let $S=\left\{v_1,v_2,v_3,\dots ,v_p\right\}$ be a linearly independent set of $p$ vectors. Then $S^{\prime }=\left\{v_1,v_2,v_3,\dots ,v_{p-1}\right\}$ is also linearly independent. So we can apply the theorem to $S^{\prime }$ and construct the vectors $T^{\prime }=\left\{u_1,u_2,u_3,\dots ,u_{p-1}\right\}$. $T^{\prime }$ is therefore an orthogonal set of nonzero vectors and $〈S^{\prime }〉=〈T^{\prime }〉$. Define

and let $T=T^{\prime }\cup \left\{u_p\right\}$. We need to now show that $T$ has several properties by building on what we know about $T^{\prime }$. But first notice that the above equation has no problems with the denominators ($〈u_i,u_i〉$) being zero, since the $u_i$ are from $T^{\prime }$, which is composed of nonzero vectors.

We show that $〈T〉=〈S〉$, by first establishing that $〈T〉\subseteq 〈S〉$. Suppose $x\in 〈T〉$, so

The term $a_p{u}_p$ is a linear combination of vectors from $T^{\prime }$ and the vector $v_p$, while the remaining terms are a linear combination of vectors from $T^{\prime }$. Since $〈T^{\prime }〉=〈S^{\prime }〉$, any term that is a multiple of a vector from $T^{\prime }$ can be rewritten as a linear combination of vectors from $S^{\prime }$. The remaining term $a_p{v}_p$ is a multiple of a vector in $S$. So we see that $x$ can be rewritten as a linear combination of vectors from $S$, i.e. $x\in 〈S〉$.

To show that $〈S〉\subseteq 〈T〉$, begin with $y\in 〈S〉$, so

Rearrange our defining equation for $u_p$ by solving for $v_p$. Then the term $a_p{v}_p$ is a multiple of a linear combination of elements of $T$. The remaining terms are a linear combination of $v_1,v_2,v_3,\dots ,v_{p-1}$, hence an element of $〈S^{\prime }〉=〈T^{\prime }〉$. Thus these remaining terms can be written as a linear combination of the vectors in $T^{\prime }$. So $y$ is a linear combination of vectors from $T$, i.e. $y\in 〈T〉$.

The elements of $T^{\prime }$ are nonzero, but what about $u_p$? Suppose to the contrary that $u_p=0$,

Since $〈S^{\prime }〉=〈T^{\prime }〉$ we can write the vectors $u_1,u_2,u_3,\dots ,u_{p-1}$ on the right side of this equation in terms of the vectors $v_1,v_2,v_3,\dots ,v_{p-1}$ and we then have the vector $v_p$ expressed as a linear combination of the other $p-1$ vectors in $S$, implying that $S$ is a linearly dependent set (Theorem DLDS), contrary to our lone hypothesis about $S$.

Finally, it is a simple matter to establish that $T$ is an orthogonal set, though it will not appear so simple looking. Think about your objects as you work through the following — what is a vector and what is a scalar. Since $T^{\prime }$ is an orthogonal set by induction, most pairs of elements in $T$ are already known to be orthogonal. We just need to test “new” inner products, between $u_p$ and $u_i$, for $1\le i\le p-1$. Here we go, using summation notation,

Example GSTV
Gram-Schmidt of three vectors
We will illustrate the Gram-Schmidt process with three vectors. Begin with the linearly independent (check this!) set

Then

and

is an orthogonal set (which you can check) of nonzero vectors and $〈T〉=〈S〉$ (all by Theorem GSP). Of course, as a by-product of orthogonality, the set $T$ is also linearly independent (Theorem OSLI). $⊠$

One final definition related to orthogonal vectors.

Definition ONS
OrthoNormal Set
Suppose $S=\left\{u_1,u_2,u_3,\dots ,u_n\right\}$ is an orthogonal set of vectors such that $\parallel u_i\parallel =1$ for all $1\le i\le n$. Then $S$ is an orthonormal set of vectors. $△$

Once you have an orthogonal set, it is easy to convert it to an orthonormal set — multiply each vector by the reciprocal of its norm, and the resulting vector will have norm 1. This scaling of each vector will not affect the orthogonality properties (apply Theorem IPSM).

Example ONTV
Orthonormal set, three vectors
The set

from Example GSTV is an orthogonal set. We compute the norm of each vector,

Converting each vector to a norm of 1, yields an orthonormal set,