Section HP  Hadamard Product

From A First Course in Linear Algebra
Version 2.20
© 2004.
Licensed under the GNU Free Documentation License.
http://linear.ups.edu/

This section is contributed by Elizabeth Million.

You may have once thought that the natural definition for matrix multiplication would be entrywise multiplication, much in the same way that a young child might say, “I writed my name.” The mistake is understandable, but it still makes us cringe. Unlike poor grammar, however, entrywise matrix multiplication has reason to be studied; it has nice properties in matrix analysis and additionally plays a role with relative gain arrays in chemical engineering, covariance matrices in probability and serves as an inertia preserver for Hermitian matrices in physics. Here we will only explore the properties of the Hadamard product in matrix analysis.

Definition HP
Hadamard Product
Let A and B be m × n matrices. The Hadamard Product of A and B is defined by A Bij = Aij Bij for all 1 i m, 1 j n.

(This definition contains Notation HP.)

As we can see, the Hadamard product is simply “entrywise multiplication”. Because of this, the Hadamard product inherits the same benefits (and restrictions) of multiplication in . Note also that both A and B need to be the same size, but not necessarily square. To avoid confusion, juxtaposition of matrices will imply the “usual” matrix multiplication, and we will use “” for the Hadamard product.

Example HP
Hadamard Product
Consider

A = 106 3 π 5 B = 313i 1 3 2 4

Then

A B = (1)(3)(0)(13)(6)(i) 3(1 3) (π)(2) (5)(4) = 3 0 6i 1 2π 20 .

Now we will explore some basics properties of the Hadamard Product.

Theorem HPC
Hadamard Product is Commutative
If A and B are m × n matrices then A B = B A.

Proof   The proof follows directly from the fact that multiplication in is commutative. Let A and B be m × n matrices. Then

A Bij = Aij Bij  Definition HP = Bij Aij  Property CMCN = B Aij  Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Definition HID
Hadamard Identity
The Hadamard identity is the m × n matrix Jmn defined by Jmn ij = 1 for all 1 i m, 1 j n.

(This definition contains Notation HID.)

Theorem HPHID
Hadamard Product with the Hadamard Identity
Suppose A is an m × n matrix. Then A Jmn = Jmn A = A.

Proof  

A Jmn ij = Jmn Aij  Theorem HPC = Jmn ij Aij  Definition HP = (1) Aij  Definition HID = Aij  Property OCN

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Definition HI
Hadamard Inverse
Let A be an m × n matrix and suppose Aij0 for all 1 i m, 1 j n. Then the Hadamard Inverse, Â , is given by Âij = (Aij)1 for all 1 i m, 1 j n.

(This definition contains Notation HI.)

Theorem HPHI
Hadamard Product with Hadamard Inverses
Let A be an m × n matrix such that Aij0 for all 1 i m, 1 j n. Then A Â = Â A = Jmn.

Proof  

A Âij = Â Aij  Theorem HPC = Âij Aij  Definition HP = (Aij)1 A ij  Definition HI Aij0 = 1  Property MICN = Jmn ij  Definition HID

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Since matrices have a different inverse and identity under the Hadamard product, we have used special notation to distinguish them from what we have been using with “normal” matrix multiplication. That is, compare “usual” matrix inverse, A1, with the Hadamard inverse Â, and the “usual” matrix identity, In, with the Hadamard identity, Jmn. The Hadamard identity matrix and the Hadamard inverse are both more limiting than helpful, so we will not explore their use further. One last fun fact for those of you who may be familiar with group theory: the set of m × n matrices with nonzero entries form an abelian (commutative) group under the Hadamard product (prove this!).

Theorem HPDAA
Hadamard Product Distributes Across Addition
Suppose A, B and C are m × n matrices. Then C (A + B) = C A + C B.

Proof  

C (A + B) ij = Cij A + Bij  Definition HP = Cij(Aij + Bij)  Definition MA = Cij Aij + Cij Bij  Property DCN = C Aij + C Bij  Definition HP = C A + C Bij  Definition MA

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Theorem HPSMM
Hadamard Product and Scalar Matrix Multiplication
Suppose α , and A and B are m × n matrices. Then α(A B) = (αA) B = A (αB).

Proof  

αA Bij = α A Bij  Definition MSM = α Aij Bij  Definition HP = αAij Bij  Definition MSM = (αA) Bij  Definition HP = α Aij Bij  Definition MSM = Aijα Bij  Property CMCN = Aij αBij  Definition MSM = A (αB) ij  Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Subsection DMHP: Diagonal Matrices and the Hadamard Product

We can relate the Hadamard product with matrix multiplication by considering diagonal matrices, since A B = AB if and only if both A and B are diagonal (Citation!!!). For example, a simple calculation reveals that the Hadamard product relates the diagonal values of a diagonalizable matrix A with its eigenvalues:

Theorem DMHP
Diagonalizable Matrices and the Hadamard Product
Let A be a diagonalizable matrix of size n with eigenvalues λ1,λ2,λ3,,λn. Let D be a diagonal matrix from the diagonalization of A, A = SDS1, and d be a vector such that Dii = di = λi for all 1 i n. Then

Aii = S (S1)td i  for all 1 i n.

That is,

A11 A22 A33 A nn = S(S1)t λ1 λ2 λ3 λ n

Proof  

S (S1)td i = k=1n S (S1)t ik dk  Definition MVP = k=1n S (S1)t ikλk  Definition of d = k=1n S ik (S1)t ikλk  Definition HP = k=1n S ik S1 kiλk  Definition TM = k=1n S ikλk S1 ki  Property CMCN = k=1n S ik Dkk S1 ki  Definition of D = j=1n k=1n S ik Dkj S1 ji Dkj = 0  for all kj = j=1n SD ij S1 ji  Theorem EMP = SDS1 ii  Theorem EMP = Aii  Definition ME

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

We obtain a similar result when we look at the singular value decomposition of square matrices (see exercises).

Theorem DMMP
Diagonal Matrices and Matrix Products
Suppose A, B are m × n matrices, and D and E are diagonal matrices of size m and n, respectively. Then,

D(A B)E = (DAE) B = (DA) (BE)

Proof  

D(A B)Eij = k=1m D ik (A B)Ekj  Theorem EMP = k=1m l=1n D ik A Bkl Elj  Theorem EMP = k=1m l=1n D ik Akl Bkl Elj  Definition HP = k=1m D ik Akj Bkj Ejj Elj = 0  for all lj = Dii Aij Bij Ejj Dik = 0  for all ik = Dii Aij Ejj Bij  Property CMCN = Dii( l=1n A il Elj) Bij Elj = 0  for all lj = Dii AEij Bij  Theorem EMP = ( k=1m D ik AEkj) Bij Dik = 0  for all ik = DAEij Bij  Theorem EMP = (DAE) Bij  Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Also,

(DAE) Bij = DAEij Bij  Definition HP = ( k=1n DA ik Ekj) Bij  Theorem EMP = DAij Ejj Bij Ekj = 0  for all kj = DAij Bij Ejj  Property CMCN = DAij( k=1n B ik Ekj) Ekj = 0  for all kj = DAij BEij  Theorem EMP = (DA) (BE) ij  Definition HP

With equality of each entry of the matrices being equal we know by Definition ME that the two matrices are equal.

Subsection EXC: Exercises

T10 Prove that A B = AB if and only if both A and B are diagonal matrices.  
Contributed by Elizabeth Million

T20 Suppose A, B are m × n matrices, and D and E are diagonal matrices of size m and n, respectively. Prove both parts of the following equality hold:

D(A B)E = (AE) (DB) = A (DBE)

 
Contributed by Elizabeth Million

T30 Let A be a square matrix of size n with singular values σ1,σ2,σ3,,σn. Let D be a diagonal matrix from the singular value decomposition of A, A = UDV (Theorem SVD). Define the vector d by di = Dii = σi, 1 i n. Prove the following equality,

Aii = (U V ¯)di

 
Contributed by Elizabeth Million

T40 Suppose A, B and C are m × n matrices. Prove that for all 1 i m,

(A B)Ct ii = (A C)Bt ii

 
Contributed by Elizabeth Million

T50 Define the diagonal matrix D of size n with entries from a vector x n by

Dij = xi if i = j 0  otherwise

Furthermore, suppose A, B are m × n matrices. Prove that ADBt ii = (A B)xi for all 1 i m.  
Contributed by Elizabeth Million