Hadamard product (matrices)

In mathematics, the Hadamard product (also known as the element-wise, entrywise[1][2]^{:ch. 5} or Schur[3] product) is a binary operation that takes two matrices of the same dimensions and produces another matrix of the same dimension as the operands, where each element i, j is the product of elements i, j of the original two matrices. It is to be distinguished from the more common matrix product. It is attributed to, and named after, either French mathematician Jacques Hadamard or German mathematician Issai Schur.

The Hadamard product operates on identically shaped matrices and produces a third matrix of the same dimensions.

The Hadamard product is associative and distributive. Unlike the matrix product, it is also commutative.[4]

Definition

For two matrices A and B of the same dimension m × n, the Hadamard product ${\displaystyle A\circ B}$ (or ${\displaystyle A\odot B}$[1][5][6][7]) is a matrix of the same dimension as the operands, with elements given by[4]

${\displaystyle (A\circ B)_{ij}=(A\odot B)_{ij}=(A)_{ij}(B)_{ij}.}$

For matrices of different dimensions (m × n and p × q, where m ≠ p or n ≠ q), the Hadamard product is undefined.

Example

For example, the Hadamard product for a 3 × 3 matrix A with a 3 × 3 matrix B is

${\displaystyle {\begin{bmatrix}a_{11}&a_{12}&a_{13}\\a_{21}&a_{22}&a_{23}\\a_{31}&a_{32}&a_{33}\end{bmatrix}}\circ {\begin{bmatrix}b_{11}&b_{12}&b_{13}\\b_{21}&b_{22}&b_{23}\\b_{31}&b_{32}&b_{33}\end{bmatrix}}={\begin{bmatrix}a_{11}\,b_{11}&a_{12}\,b_{12}&a_{13}\,b_{13}\\a_{21}\,b_{21}&a_{22}\,b_{22}&a_{23}\,b_{23}\\a_{31}\,b_{31}&a_{32}\,b_{32}&a_{33}\,b_{33}\end{bmatrix}}.}$

Properties

• The Hadamard product is commutative (when working with a commutative ring), associative and distributive over addition. That is, if A, B, and C are matrices of the same size, and k is a scalar:

  ${\displaystyle {\begin{aligned}&\mathbf {A} \circ \mathbf {B} =\mathbf {B} \circ \mathbf {A} ,\\&\mathbf {A} \circ (\mathbf {B} \circ \mathbf {C} )=(\mathbf {A} \circ \mathbf {B} )\circ \mathbf {C} ,\\&\mathbf {A} \circ (\mathbf {B} +\mathbf {C} )=\mathbf {A} \circ \mathbf {B} +\mathbf {A} \circ \mathbf {C} ,\\&\left(k\mathbf {A} \right)\circ \mathbf {B} =\mathbf {A} \circ \left(k\mathbf {B} \right)=k\left(\mathbf {A} \circ \mathbf {B} \right),\\&\mathbf {A} \circ \mathbf {0} =\mathbf {0} \circ \mathbf {A} =\mathbf {0} .\end{aligned}}}$

• The identity matrix under Hadamard multiplication of two m × n matrices is an m × n matrix where all elements are equal to 1. This is different from the identity matrix under regular matrix multiplication, where only the elements of the main diagonal are equal to 1. Furthermore, a matrix has an inverse under Hadamard multiplication if and only if none of the elements are equal to zero.[8]
• For vectors x and y, and corresponding diagonal matrices D_x and D_y with these vectors as their main diagonals, the following identity holds:[2]^:479

  ${\displaystyle \mathbf {x} ^{*}(\mathbf {A} \circ \mathbf {B} )\mathbf {y} =\mathrm {tr} \left(\mathbf {D} _{\mathbf {x} }^{*}\mathbf {A} \mathbf {D} _{\mathbf {y} }\mathbf {B} ^{\mathsf {T}}\right),}$

  where x* denotes the conjugate transpose of x. In particular, using vectors of ones, this shows that the sum of all elements in the Hadamard product is the trace of AB^T. A related result for square A and B, is that the row-sums of their Hadamard product are the diagonal elements of AB^T:[9]

  ${\displaystyle {\begin{aligned}\sum _{i}(A\circ B)_{ij}&=\left(B^{\mathsf {T}}A\right)_{jj}\\&=\left(AB^{\mathsf {T}}\right)_{ii}.\end{aligned}}}$

  Similarly

  ${\displaystyle \left(\mathbf {y} \mathbf {x} ^{*}\right)\circ \mathbf {A} =\mathbf {D} _{\mathbf {y} }\mathbf {A} \mathbf {D} _{\mathbf {x} }^{*}}$

• The Hadamard product is a principal submatrix of the Kronecker product.
• The Hadamard product satisfies the rank inequality

  ${\displaystyle \operatorname {rank} (\mathbf {A} \circ \mathbf {B} )\leq \operatorname {rank} (\mathbf {A} )\operatorname {rank} (\mathbf {B} )}$

• If A and B are positive-definite matrices, then the following inequality involving the Hadamard product is valid:[10]

  ${\displaystyle \prod _{i=k}^{n}\lambda _{i}(\mathbf {A} \circ \mathbf {B} )\geq \prod _{i=k}^{n}\lambda _{i}(\mathbf {A} \mathbf {B} ),\quad k=1,\ldots ,n,}$

  where λ_i(A) is the ith largest eigenvalue of A.
• If D and E are diagonal matrices, then[11]

  ${\displaystyle {\begin{aligned}\mathbf {D} (\mathbf {A} \circ \mathbf {B} )\mathbf {E} &=(\mathbf {D} \mathbf {A} \mathbf {E} )\circ \mathbf {B} =(\mathbf {D} \mathbf {A} )\circ (\mathbf {B} \mathbf {E} )\\&=(\mathbf {A} \mathbf {E} )\circ (\mathbf {D} \mathbf {B} )=\mathbf {A} \circ (\mathbf {D} \mathbf {B} \mathbf {E} ).\end{aligned}}}$

• The Hadamard product of two vectors ${\displaystyle \mathbf {a} }$ and ${\displaystyle \mathbf {b} }$ is the same as matrix multiplication of one vector by the corresponding diagonal matrix of the other vector:

  ${\displaystyle \mathbf {a} \circ \mathbf {b} =\mathbf {D} _{\mathbf {a} }\mathbf {b} .}$

The mixed-product property

${\displaystyle (A\otimes B)\circ (C\otimes D)=(A\circ C)\otimes (B\circ D)}$, where ${\displaystyle \otimes }$ is Kronecker product

${\displaystyle (A\bullet B)\circ (C\bullet D)=(A\circ C)\bullet (B\circ D)}$, where ${\displaystyle \bullet }$ denotes Face-splitting product.[12]

${\displaystyle (A\bullet B)(C\ast D)=(AC)\circ (BD)}$, where ${\displaystyle \ast }$ is column-wise Khatri–Rao product.

Schur product theorem

The Hadamard product of two positive-semidefinite matrices is positive-semidefinite.[4][9] This is known as the Schur product theorem,[8] after Russian mathematician Issai Schur. For two positive-semidefinite matrices A and B, it is also known that the determinant of their Hadamard product is greater than or equal to the product of their respective determinants:[9]

${\displaystyle \det(\mathbf {A} \circ \mathbf {B} )\geq \det(\mathbf {A} )\det(\mathbf {B} ).}$

In programming languages

Hadamard multiplication is built into certain programming languages under various names. In MATLAB, GNU Octave, GAUSS and HP Prime, it is known as array multiplication, or in Julia broadcast multiplication, with the symbol .*.[13] In Fortran, R,[14] APL, J and Wolfram Language (Mathematica), it is done through simple multiplication operator *, whereas the matrix product is done through the function matmul, %*%, +.×, +/ .* and the . operators, respectively. In Python with the NumPy numerical library or the SymPy symbolic library, multiplication of array objects as a1*a2 produces the Hadamard product, but otherwise multiplication as a1@a2 or matrix objects m1*m2 will produce a matrix product. The Eigen C++ library provides a cwiseProduct member function for the Matrix class (a.cwiseProduct(b)), while the Armadillo library uses the operator % to make compact expressions (a % b; a * b is a matrix product).

Applications

The Hadamard product appears in lossy compression algorithms such as JPEG. The decoding step involves an entry-for-entry product, in other words the Hadamard product.

It is also used in the machine learning literature, for example to describe the architecture of recurrent neural networks as GRUs or LSTMs.

Analogous operations

Other Hadamard operations are also seen in the mathematical literature,[15] namely the Hadamard root and Hadamard power (which are in effect the same thing because of fractional indices), defined for a matrix such that:

For

${\displaystyle {\begin{aligned}\mathbf {B} &=\mathbf {A} ^{\circ 2}\\B_{ij}&={A_{ij}}^{2}\end{aligned}}}$

and for

${\displaystyle {\begin{aligned}\mathbf {B} &=\mathbf {A} ^{\circ {\frac {1}{2}}}\\B_{ij}&={A_{ij}}^{\frac {1}{2}}\end{aligned}}}$

The Hadamard inverse reads:[15]

${\displaystyle {\begin{aligned}\mathbf {B} &=\mathbf {A} ^{\circ -1}\\B_{ij}&={A_{ij}}^{-1}\end{aligned}}}$

A Hadamard division is defined as:[16][17]

${\displaystyle {\begin{aligned}\mathbf {C} &=\mathbf {A} \oslash \mathbf {B} \\C_{ij}&={\frac {A_{ij}}{B_{ij}}}\end{aligned}}}$

The penetrating face product

The penetrating face product of matrices

According to the definition of V. Slyusar the penetrating face product of the pxg matrix ${\displaystyle \mathbf {A} }$ and n-dimensional matrix ${\displaystyle \mathbf {B} }$ (n > 1) that is unfolded in the block of rows or block of columns with pxg blocks (${\displaystyle \mathbf {B} =[B_{n}]}$) is a matrix of size ${\displaystyle \mathbf {B} }$ of the form:[18]

${\displaystyle \mathbf {A} [\circ ]\mathbf {B} =\left[{\begin{array}{c | c | c }\mathbf {A} \circ \mathbf {B} _{1}&\mathbf {A} \circ \mathbf {B} _{2}&\mathbf {A} \circ \mathbf {B} _{3}\end{array}}\right]}$.

Example

If

${\displaystyle \mathbf {A} =\left[{\begin{array}{c}1&2&3\\4&5&6\\7&8&9\end{array}}\right],\quad \mathbf {B} =\left[{\begin{array}{c | c | c }\mathbf {B} _{1}&\mathbf {B} _{2}&\mathbf {B} _{3}\end{array}}\right]=\left[{\begin{array}{c c c | c c c | c c c }1&4&7&2&8&14&3&12&21\\8&20&5&10&25&40&12&30&6\\2&8&3&2&4&2&7&3&9\end{array}}\right]}$

then

${\displaystyle \mathbf {A} [\circ ]\mathbf {B} =\left[{\begin{array}{c c c | c c c | c c c }1&8&21&2&16&42&3&24&63\\32&100&30&40&125&240&48&150&36\\14&64&27&14&32&18&49&24&81\end{array}}\right]}$.

Main properties

${\displaystyle \mathbf {A} [\circ ]\mathbf {B} =\mathbf {B} [\circ ]\mathbf {A} }$;[18]

${\displaystyle \mathbf {M} \bullet \mathbf {M} =\mathbf {M} [\circ ](\mathbf {M} \otimes \mathbf {1} ^{\textsf {T}})}$,

where ${\displaystyle \bullet }$ denotes the Face-splitting product of matrices,

${\displaystyle \mathbf {c} \bullet \mathbf {M} =\mathbf {c} [\circ ]\mathbf {M} }$, where ${\displaystyle \mathbf {c} }$ is a vector.

Applications

The penetrating face product is used in the tensor-matrix theory of digital antenna arrays.[18] This operation can also be used in artificial neural network models, specifically convolutional layers.

See also

• Frobenius inner product
• Pointwise product
• Kronecker product
• Khatri–Rao product

References

1. "Comprehensive List of Algebra Symbols". Math Vault. 2020-03-25. Retrieved 2020-09-06.
2. Horn, Roger A.; Johnson, Charles R. (2012). Matrix analysis. Cambridge University Press.
3. Davis, Chandler (1962). "The norm of the Schur product operation". Numerische Mathematik. 4 (1): 343–44. doi:10.1007/bf01386329.
4. Million, Elizabeth (April 12, 2007). "The Hadamard Product" (PDF). buzzard.ups.edu. Retrieved September 6, 2020.
5. "Hadamard product - Machine Learning Glossary". machinelearning.wtf.
6. "linear algebra - What does a dot in a circle mean?". Mathematics Stack Exchange.
7. "Element-wise (or pointwise) operations notation?". Mathematics Stack Exchange.
8. Million, Elizabeth. "The Hadamard Product" (PDF). Retrieved 2 January 2012.
9. Styan, George P. H. (1973), "Hadamard Products and Multivariate Statistical Analysis", Linear Algebra and Its Applications, 6: 217–240, doi:10.1016/0024-3795(73)90023-2, hdl:10338.dmlcz/102190
10. Hiai, Fumio; Lin, Minghua (February 2017). "On an eigenvalue inequality involving the Hadamard product". Linear Algebra and Its Applications. 515: 313–320. doi:10.1016/j.laa.2016.11.017.
11. "Project" (PDF). buzzard.ups.edu. 2007. Retrieved 2019-12-18.
12. Slyusar, V. I. (December 27, 1996). "End products in matrices in radar applications" (PDF). Radioelectronics and Communications Systems.– 1998, Vol. 41; Number 3: 50–53.
13. "Arithmetic Operators + - * / \ ^ ' -". MATLAB documentation. MathWorks. Archived from the original on 24 April 2012. Retrieved 2 January 2012.
14. "Matrix multiplication". An Introduction to R. The R Project for Statistical Computing. 16 May 2013. Retrieved 24 August 2013.
15. Reams, Robert (1999). "Hadamard inverses, square roots and products of almost semidefinite matrices". Linear Algebra and Its Applications. 288: 35–43. doi:10.1016/S0024-3795(98)10162-3.
16. Wetzstein, Gordon; Lanman, Douglas; Hirsch, Matthew; Raskar, Ramesh. "Supplementary Material: Tensor Displays: Compressive Light Field Synthesis using Multilayer Displays with Directional Backlighting" (PDF). MIT Media Lab.
17. Cyganek, Boguslaw (2013). Object Detection and Recognition in Digital Images: Theory and Practice. John Wiley & Sons. p. 109. ISBN 9781118618363.
18. Slyusar, V. I. (March 13, 1998). "A Family of Face Products of Matrices and its properties" (PDF). Cybernetics and Systems Analysis C/C of Kibernetika I Sistemnyi Analiz. 1999. 35 (3): 379–384. doi:10.1007/BF02733426.