Section HP Hadamard Product

From A First Course in Linear Algebra
Version 2.11
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

This section is contributed by Elizabeth Million.

You may have once thought that the natural definition for matrix multiplication would be entrywise multiplication, much in the same way that a young child might say, “I writed my name.” The mistake is understandable, but it still makes us cringe. Unlike poor grammar, however, entrywise matrix multiplication has reason to be studied; it has nice properties in matrix analysis and additionally plays a role with relative gain arrays in chemical engineering, covariance matrices in probability and serves as an inertia preserver for Hermitian matrices in physics. Here we will only explore the properties of the Hadamard product in matrix analysis.

Definition HP
Hadamard Product
Let $A$ and $B$ be $m \times n$ matrices. The Hadamard Product of $A$ and $B$ is defined by ${[A \circ B]}_{i j} = {[A]}_{i j} {[B]}_{i j}$ for all $1 \leq i \leq m$ , $1 \leq j \leq n$ .

(This definition contains Notation HP.) $△$

As we can see, the Hadamard product is simply “entrywise multiplication”. Because of this, the Hadamard product inherits the same benefits (and restrictions) of multiplication in $ℂ$ . Note also that both $A$ and $B$ need to be the same size, but not necessarily square. To avoid confusion, juxtaposition of matrices will imply the “usual” matrix multiplication, and we will use “ $\circ$ ” for the Hadamard product.

Example HP
Hadamard Product
Consider

\begin{array}{l} A = [\begin{matrix} 1 & 0 & 6 \\ 3 & π & 5 \end{matrix}] & B = [\begin{matrix} 3 & 13 & i \\ \frac{1}{3} & 2 & 4 \end{matrix}] \end{array}

Then

\begin{array}{l} A \circ B & = [\begin{matrix} (1) (3) & (0) (13) & (6) (i) \\ 3 (\frac{1}{3}) & (π) (2) & (5) (4) \end{matrix}] \\ = [\begin{matrix} 3 & 0 & 6 i \\ 1 & 2 π & 20 \end{matrix}] . \end{array}

⊠

Now we will explore some basics properties of the Hadamard Product.

Theorem HPC
Hadamard Product is Commutative
If $A$ and $B$ are $m \times n$ matrices then $A \circ B = B \circ A$ . $□$

Proof The proof follows directly from the fact that multiplication in $ℂ$ is commutative. Let $A$ and $B$ be $m \times n$ matrices. Then

\begin{array}{l} {[A \circ B]}_{i j} & = {[A]}_{i j} {[B]}_{i j} & Definition HP \\ = {[B]}_{i j} {[A]}_{i j} & Property CMCN \\ = {[B \circ A]}_{i j} & Definition HP \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

Definition HID
Hadamard Identity
The Hadamard identity is the $m \times n$ matrix $J m n$ defined by ${[J m n]}_{i j} = 1$ for all $1 \leq i \leq m$ , $1 \leq j \leq n$ .

(This definition contains Notation HID.) $△$

Theorem HPHID
Hadamard Product with the Hadamard Identity
Suppose $A$ is an $m \times n$ matrix. Then $A \circ J m n = J m n \circ A = A$ . $□$

Proof

\begin{array}{l} {[A \circ J m n]}_{i j} & = {[J m n \circ A]}_{i j} & Theorem HPC \\ = {[J m n]}_{i j} {[A]}_{i j} & Definition HP \\ = (1) {[A]}_{i j} & Definition HID \\ = {[A]}_{i j} & Property OCN \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

Definition HI
Hadamard Inverse
Let $A$ be an $m \times n$ matrix and suppose ${[A]}_{i j} \neq 0$ for all $1 \leq i \leq m$ , $1 \leq j \leq n$ . Then the Hadamard Inverse, $\hat{A}$ , is given by ${[\hat{A}]}_{i j} = {({[A]}_{i j})}^{- 1}$ for all $1 \leq i \leq m$ , $1 \leq j \leq n$ .

(This definition contains Notation HI.) $△$

Theorem HPHI
Hadamard Product with Hadamard Inverses
Let $A$ be an $m \times n$ matrix such that ${[A]}_{i j} \neq 0$ for all $1 \leq i \leq m$ , $1 \leq j \leq n$ . Then $A \circ \hat{A} = \hat{A} \circ A = J m n$ . $□$

Proof

\begin{array}{l} {[A \circ \hat{A}]}_{i j} & = {[\hat{A} \circ A]}_{i j} & Theorem HPC \\ = {[\hat{A}]}_{i j} {[A]}_{i j} & Definition HP \\ = {({[A]}_{i j})}^{- 1} {[A]}_{i j} & Definition HI, {[A]}_{i j} \neq 0 \\ = 1 & Property MICN \\ = {[J m n]}_{i j} & Definition HID \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

Since matrices have a different inverse and identity under the Hadamard product, we have used special notation to distinguish them from what we have been using with “normal” matrix multiplication. That is, compare “usual” matrix inverse, $A^{- 1}$ , with the Hadamard inverse $\hat{A}$ , and the “usual” matrix identity, $I_{n}$ , with the Hadamard identity, $J m n$ . The Hadamard identity matrix and the Hadamard inverse are both more limiting than helpful, so we will not explore their use further. One last fun fact for those of you who may be familiar with group theory: the set of $m \times n$ matrices with nonzero entries form an abelian (commutative) group under the Hadamard product (prove this!).

Theorem HPDAA
Hadamard Product Distributes Across Addition
Suppose $A$ , $B$ and $C$ are $m \times n$ matrices. Then $C \circ (A + B) = C \circ A + C \circ B$ . $□$

Proof

\begin{array}{l} {[C \circ (A + B)]}_{i j} & = {[C]}_{i j} {[A + B]}_{i j} & Definition HP \\ = {[C]}_{i j} ({[A]}_{i j} + {[B]}_{i j}) & Definition MA \\ = {[C]}_{i j} {[A]}_{i j} + {[C]}_{i j} {[B]}_{i j} & Property DCN \\ = {[C \circ A]}_{i j} + {[C \circ B]}_{i j} & Definition HP \\ = {[C \circ A + C \circ B]}_{i j} & Definition MA \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

Theorem HPSMM
Hadamard Product and Scalar Matrix Multiplication
Suppose $α \in ℂ$ , and $A$ and $B$ are $m \times n$ matrices. Then $α (A \circ B) = (α A) \circ B = A \circ (α B)$ . $□$

Proof

\begin{array}{l} {[α A \circ B]}_{i j} & = α {[A \circ B]}_{i j} & Definition MSM \\ = α {[A]}_{i j} {[B]}_{i j} & Definition HP \\ = {[α A]}_{i j} {[B]}_{i j} & Definition MSM \\ = {[(α A) \circ B]}_{i j} & Definition HP \\ = α {[A]}_{i j} {[B]}_{i j} & Definition MSM \\ = {[A]}_{i j} α {[B]}_{i j} & Property CMCN \\ = {[A]}_{i j} {[α B]}_{i j} & Definition MSM \\ = {[A \circ (α B)]}_{i j} & Definition HP \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

Subsection DMHP: Diagonal Matrices and the Hadamard Product

We can relate the Hadamard product with matrix multiplication by considering diagonal matrices, since $A \circ B = A B$ if and only if both $A$ and $B$ are diagonal (Citation!!!). For example, a simple calculation reveals that the Hadamard product relates the diagonal values of a diagonalizable matrix $A$ with its eigenvalues:

Theorem DMHP
Diagonalizable Matrices and the Hadamard Product
Let $A$ be a diagonalizable matrix of size $n$ with eigenvalues $λ_{1}, λ_{2}, λ_{3}, \dots, λ_{n}$ . Let $D$ be a diagonal matrix from the diagonalization of $A$ , $A = S D S^{- 1}$ , and $d$ be a vector such that ${[D]}_{i i}$ $=$ ${[d]}_{i}$ $= λ_{i}$ for all $1 \leq i \leq n$ . Then

\begin{array}{l} {[A]}_{i i} & = {[S \circ {(S^{- 1})}^{t} d]}_{i} & for all 1 \leq i \leq n . \end{array}

That is,

[\begin{matrix} {[A]}_{11} \\ {[A]}_{22} \\ {[A]}_{33} \\ ⋮ \\ {[A]}_{n n} \end{matrix}] = S \circ {(S^{- 1})}^{t} [\begin{matrix} λ_{1} \\ λ_{2} \\ λ_{3} \\ ⋮ \\ λ_{n} \end{matrix}]

□

Proof

\begin{array}{l} {[S \circ {(S^{- 1})}^{t} d]}_{i} & = \sum_{k = 1}^{n} {[S \circ {(S^{- 1})}^{t}]}_{i k} {[d]}_{k} & Definition MVP \\ = \sum_{k = 1}^{n} {[S \circ {(S^{- 1})}^{t}]}_{i k} λ_{k} & Definition of d \\ = \sum_{k = 1}^{n} {[S]}_{i k} {[{(S^{- 1})}^{t}]}_{i k} λ_{k} & Definition HP \\ = \sum_{k = 1}^{n} {[S]}_{i k} {[S^{- 1}]}_{k i} λ_{k} & Definition TM \\ = \sum_{k = 1}^{n} {[S]}_{i k} λ_{k} {[S^{- 1}]}_{k i} & Property CMCN \\ = \sum_{k = 1}^{n} {[S]}_{i k} {[D]}_{k k} {[S^{- 1}]}_{k i} & Definition of D \\ = \sum_{j = 1}^{n} \sum_{k = 1}^{n} {[S]}_{i k} {[D]}_{k j} {[S^{- 1}]}_{j i} & {[D]}_{k j} = 0 for all k \neq j \\ = \sum_{j = 1}^{n} {[S D]}_{i j} {[S^{- 1}]}_{j i} & Theorem EMP \\ = {[S D S^{- 1}]}_{i i} & Theorem EMP \\ = {[A]}_{i i} & Definition ME \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

We obtain a similar result when we look at the singular value decomposition of square matrices (see exercises).

Theorem DMMP
Diagonal Matrices and Matrix Products
Suppose $A$ , $B$ are $m \times n$ matrices, and $D$ and $E$ are diagonal matrices of size $m$ and $n$ , respectively. Then,

D (A \circ B) E = (D A E) \circ B = (D A) \circ (B E)

□

Proof

\begin{array}{l} {[D (A \circ B) E]}_{i j} & = \sum_{k = 1}^{m} {[D]}_{i k} {[(A \circ B) E]}_{k j} & Theorem EMP \\ = \sum_{k = 1}^{m} \sum_{l = 1}^{n} {[D]}_{i k} {[A \circ B]}_{k l} {[E]}_{l j} & Theorem EMP \\ = \sum_{k = 1}^{m} \sum_{l = 1}^{n} {[D]}_{i k} {[A]}_{k l} {[B]}_{k l} {[E]}_{l j} & Definition HP \\ = \sum_{k = 1}^{m} {[D]}_{i k} {[A]}_{k j} {[B]}_{k j} {[E]}_{j j} & {[E]}_{l j} = 0 for all l \neq j \\ = {[D]}_{i i} {[A]}_{i j} {[B]}_{i j} {[E]}_{j j} & {[D]}_{i k} = 0 for all i \neq k \\ = {[D]}_{i i} {[A]}_{i j} {[E]}_{j j} {[B]}_{i j} & Property CMCN \\ = {[D]}_{i i} (\sum_{l = 1}^{n} {[A]}_{i l} {[E]}_{l j}) {[B]}_{i j} & {[E]}_{l j} = 0 for all l \neq j \\ = {[D]}_{i i} {[A E]}_{i j} {[B]}_{i j} & Theorem EMP \\ = (\sum_{k = 1}^{m} {[D]}_{i k} {[A E]}_{k j}) {[B]}_{i j} & {[D]}_{i k} = 0 for all i \neq k \\ = {[D A E]}_{i j} {[B]}_{i j} & Theorem EMP \\ = {[(D A E) \circ B]}_{i j} & Definition HP \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Also,

\begin{array}{l} {[(D A E) \circ B]}_{i j} & = {[D A E]}_{i j} {[B]}_{i j} & Definition HP \\ = (\sum_{k = 1}^{n} {[D A]}_{i k} {[E]}_{k j}) {[B]}_{i j} & Theorem EMP \\ = {[D A]}_{i j} {[E]}_{j j} {[B]}_{i j} & {[E]}_{k j} = 0 for all k \neq j \\ = {[D A]}_{i j} {[B]}_{i j} {[E]}_{j j} & Property CMCN \\ = {[D A]}_{i j} (\sum_{k = 1}^{n} {[B]}_{i k} {[E]}_{k j}) & {[E]}_{k j} = 0 for all k \neq j \\ = {[D A]}_{i j} {[B E]}_{i j} & Theorem EMP \\ = {[(D A) \circ (B E)]}_{i j} & Definition HP \end{array}

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal. $■$

Subsection EXC: Exercises

T10 Prove that $A \circ B = A B$ if and only if both $A$ and $B$ are diagonal matrices.
Contributed by Elizabeth Million

T20 Suppose $A$ , $B$ are $m \times n$ matrices, and $D$ and $E$ are diagonal matrices of size $m$ and $n$ , respectively. Prove both parts of the following equality hold:

\begin{array}{l} D (A \circ B) E & = (A E) \circ (D B) = A \circ (D B E) \end{array}

Contributed by Elizabeth Million

T30 Let $A$ be a square matrix of size $n$ with singular values $σ_{1}, σ_{2}, σ_{3}, \dots, σ_{n}$ . Let $D$ be a diagonal matrix from the singular value decomposition of $A$ , $A = U D V^{*}$ (Theorem SVD). Define the vector $d$ by ${[d]}_{i} = {[D]}_{i i} = σ_{i}$ , $1 \leq i \leq n$ . Prove the following equality,

\begin{array}{l} {[A]}_{i i} & = {[(U \circ \bar{V}) d]}_{i} \end{array}

Contributed by Elizabeth Million

T40 Suppose $A$ , $B$ and $C$ are $m \times n$ matrices. Prove that for all $1 \leq i \leq m$ ,

\begin{array}{l} {[(A \circ B) C^{t}]}_{i i} & = {[(A \circ C) B^{t}]}_{i i} \end{array}

Contributed by Elizabeth Million

T50 Define the diagonal matrix $D$ of size $n$ with entries from a vector $x \in ℂ^{n}$ by

\begin{array}{l} {[D]}_{i j} & = \{\begin{matrix} {[x]}_{i} & if i = j \\ 0 & otherwise \end{matrix} \end{array}

Furthermore, suppose $A$ , $B$ are $m \times n$ matrices. Prove that ${[A D B^{t}]}_{i i} = {[(A \circ B) x]}_{i}$ for all $1 \leq i \leq m$ .
Contributed by Elizabeth Million

[next] [prev] [prev-tail] [front] [up]