13.7 Products Entry by Entry

Matrix multiplication is not the only way to multiply two matrices of the same size. There is a much simpler product — multiply the entries in matching positions — which is useless for composing linear maps and extremely useful everywhere else. This section introduces it and proves the one theorem that makes it interesting: it preserves positivity. A determinant inequality follows, and it contains Hadamard’s inequality of Section 6 as a special case.

Throughout, $F = \mathbb{R}$ or $F = \mathbb{C}$ .

13.7.1. The entrywise product

Two matrices of the same shape can be multiplied the way two lists of numbers are: position by position. The operation has no interpretation in terms of linear maps, which is one reason it looks like a curiosity at first sight, and it is written with a small circle to keep it away from ordinary multiplication.

The Hadamard product multiplies matrices the way a spreadsheet would.

Definition 13.7.1 (Hadamard Product)

Let $\mathbf{A}, \mathbf{B} \in M_{m \times n}(F)$ . Their Hadamard product, or entrywise product, is the matrix $\mathbf{A} \circ \mathbf{B} \in M_{m \times n}(F)$ with

$(\mathbf{A} \circ \mathbf{B})_{ij} = a_{ij}b_{ij} \qquad\text{for all } i, j .$

Both factors must have the same shape, and the result has that shape too; there is no rule like “columns of the first match rows of the second”. The operation is commutative and associative, and it is bilinear — $(\mathbf{A} + \mathbf{A}') \circ \mathbf{B} = \mathbf{A} \circ \mathbf{B} + \mathbf{A}' \circ \mathbf{B}$ and $(c\mathbf{A}) \circ \mathbf{B} = c(\mathbf{A} \circ \mathbf{B})$ — because each of these is a statement about numbers, checked one entry at a time. Its identity element is the all-ones matrix $\mathbf{J}$ , not $\mathbf{I}$ .

Examples.

$\begin{pmatrix} 1 & 2 \\ 3 & 4\end{pmatrix} \circ \begin{pmatrix} 5 & 0 \\ -1 & 2\end{pmatrix} = \begin{pmatrix} 5 & 0 \\ -3 & 8\end{pmatrix}$ .
$\mathbf{A} \circ \mathbf{I} = \operatorname{diag}(a_{11}, \dots, a_{nn})$ : multiplying entrywise by the identity erases everything off the diagonal, where ordinary multiplication by $\mathbf{I}$ changes nothing.
$\mathbf{A} \circ \mathbf{J} = \mathbf{A}$ , so $\mathbf{J}$ is the identity for $\circ$ .
For two diagonal matrices, $\mathbf{D} \circ \mathbf{D}' = \mathbf{D}\mathbf{D}'$ : both products are $\operatorname{diag}(d_1d_1', \dots, d_nd_n')$ . For matrices in general the two have nothing to do with each other, though they can coincide by accident: $\begin{psmallmatrix}1&1\\mathbf{0}&0\end{psmallmatrix}$ and $\begin{psmallmatrix}1&0\\mathbf{0}&0\end{psmallmatrix}$ have equal Hadamard and matrix products without either being diagonal.
For column vectors $\mathbf{u}, \mathbf{w} \in F^n$ , regarded as $n \times 1$ matrices, $\mathbf{u} \circ \mathbf{w} = (u_1w_1, \dots, u_nw_n)$ .

Non-example by minimal change. Take $\mathbf{A} = \begin{psmallmatrix} 0 & 1 \\ 1 & 0\end{psmallmatrix}$ and $\mathbf{B} = \mathbf{A}$ . Then $\mathbf{A}\mathbf{B} = \mathbf{I}$ , but $\mathbf{A} \circ \mathbf{B} = \begin{psmallmatrix} 0 & 1 \\ 1 & 0\end{psmallmatrix} \circ \begin{psmallmatrix} 0 & 1 \\ 1 & 0\end{psmallmatrix} = \begin{psmallmatrix} 0 & 1 \\ 1 & 0\end{psmallmatrix}$ . The two products differ, and they differ in the most basic way: $\mathbf{A}$ is its own inverse and its own Hadamard square.

Warning. $\mathbf{A} \circ \mathbf{B}$ is not $\mathbf{A}\mathbf{B}$ , and almost nothing carries over. Two invertible matrices can have a singular entrywise product:

$\begin{pmatrix} 1 & 2 \\ 2 & 1\end{pmatrix} \circ \begin{pmatrix} 2 & 1 \\ 1 & 2\end{pmatrix} = \begin{pmatrix} 2 & 2 \\ 2 & 2\end{pmatrix} ,$

where the two factors have determinants $-3$ and $3$ and the product has determinant $0$ . There is no formula for $\det(\mathbf{A} \circ \mathbf{B})$ in terms of $\det \mathbf{A}$ and $\det \mathbf{B}$ , no relation between the eigenvalues, and no rule $(\mathbf{A} \circ \mathbf{B})^{-1} = \mathbf{A}^{-1} \circ \mathbf{B}^{-1}$ . The one structural property that does survive is the subject of this section, and it is a genuine surprise.

The name is the same one that Section 6 attached to a determinant inequality. That is a coincidence of a busy mathematician, not of the mathematics: Hadamard’s inequality and the Hadamard product are unrelated statements, and this section will nonetheless prove one from the other.

13.7.2. The Schur product theorem

Positivity is a statement about the quadratic form $\mathbf{x}^{*}\mathbf{A}\mathbf{x}$ , and the quadratic form of $\mathbf{A} \circ \mathbf{B}$ has no visible relation to those of $\mathbf{A}$ and $\mathbf{B}$ . So the following theorem cannot be proved by staring at the form. What works is to break both matrices into the simplest positive pieces there are — rank-one matrices $\mathbf{u}\mathbf{u}^{*}$ — and to notice that the entrywise product of two such pieces is again one.

Lemma 13.7.1 (Entrywise product of two rank-one positive matrices)

Let $\mathbf{u}, \mathbf{w} \in F^n$ . Then

$(\mathbf{u}\mathbf{u}^{*}) \circ (\mathbf{w}\mathbf{w}^{*}) = (\mathbf{u} \circ \mathbf{w})(\mathbf{u} \circ \mathbf{w})^{*} .$

Proof. Compare the $(i, j)$ entries. The $(i, j)$ entry of $\mathbf{u}\mathbf{u}^{*}$ is $u_i\overline{u_j}$ and that of $\mathbf{w}\mathbf{w}^{*}$ is $w_i\overline{w_j}$ , so the left-hand side has $(i, j)$ entry

$u_i\overline{u_j}\,w_i\overline{w_j} = (u_iw_i)\,\overline{(u_jw_j)} ,$

the rearrangement using commutativity of $F$ and $\overline{u_j}\,\overline{w_j} = \overline{u_jw_j}$ . The vector $\mathbf{u} \circ \mathbf{w}$ has $i$ -th entry $u_iw_i$ , so the right-hand side has $(i, j)$ entry $(u_iw_i)\overline{(u_jw_j)}$ as well. The two matrices agree entry by entry.

That one line is the whole theorem; everything else is bookkeeping over the spectral decompositions.

Theorem 13.7.2 (Schur Product Theorem)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ .

(a) If $\mathbf{A} \succeq 0$ and $\mathbf{B} \succeq 0$ , then $\mathbf{A} \circ \mathbf{B} \succeq 0$ .
(b) If $\mathbf{A} \succ 0$ and $\mathbf{B} \succ 0$ , then $\mathbf{A} \circ \mathbf{B} \succ 0$ .

Idea. For (a): the spectral theorem writes $\mathbf{A} = \sum_i\lambda_i\mathbf{u}_i\mathbf{u}_i^{*}$ and $\mathbf{B} = \sum_j\mu_j\mathbf{w}_j\mathbf{w}_j^{*}$ with all $\lambda_i, \mu_j \ge 0$ . Expand $\mathbf{A} \circ \mathbf{B}$ by bilinearity into $n^2$ terms; by Lemma 13.7.1 each term is a non-negative multiple of a matrix $\mathbf{z}\mathbf{z}^{*}$ , and those are positive semidefinite. A sum of positive semidefinite matrices is positive semidefinite, so we are done.

For (b), do not redo the argument with strict inequalities: the vectors $\mathbf{u}_i \circ \mathbf{w}_j$ need not span, and the bookkeeping becomes unpleasant. Instead peel a multiple of the identity off $\mathbf{B}$ . If $\varepsilon > 0$ is the smallest eigenvalue of $\mathbf{B}$ , then $\mathbf{B} = (\mathbf{B} - \varepsilon\mathbf{I}) + \varepsilon\mathbf{I}$ with $\mathbf{B} - \varepsilon\mathbf{I} \succeq 0$ , so $\mathbf{A} \circ \mathbf{B} = \mathbf{A} \circ (\mathbf{B} - \varepsilon\mathbf{I}) + \varepsilon\,(\mathbf{A} \circ \mathbf{I}) ,$ and $\mathbf{A} \circ \mathbf{I}$ is the diagonal matrix of the $a_{ii}$ , which are strictly positive. Part (a) handles the first term, and a positive definite matrix plus a positive semidefinite one is positive definite.

Proof.

(a) Both $\mathbf{A}$ and $\mathbf{B}$ are Hermitian, so each has an orthonormal basis of eigenvectors. Let $(\mathbf{u}_1, \dots, \mathbf{u}_n)$ be one for $\mathbf{A}$ , with $\mathbf{A}\mathbf{u}_i = \lambda_i\mathbf{u}_i$ , and let $\mathbf{U}$ be the unitary matrix with these columns, so that $\mathbf{A} = \mathbf{U}\mathbf{D}\mathbf{U}^{*}$ with $\mathbf{D} = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$ by Corollary 12.4.3 over $\mathbb{C}$ and Corollary 12.5.2 over $\mathbb{R}$ . The columns of $\mathbf{U}\mathbf{D}$ are $\lambda_i\mathbf{u}_i$ and the rows of $\mathbf{U}^{*}$ are $\mathbf{u}_i^{*}$ , so Corollary 8.1.3 gives
$\mathbf{A} = \sum_{i=1}^{n}\lambda_i\,\mathbf{u}_i\mathbf{u}_i^{*} ,$

which over $\mathbb{R}$ is Corollary 12.5.4. In the same way $\mathbf{B} = \sum_{j}\mu_j\mathbf{w}_j\mathbf{w}_j^{*}$ . Every $\lambda_i \ge 0$ and every $\mu_j \ge 0$ , by Theorem 13.1.4 (b).

The Hadamard product is bilinear, so expanding both sums,

$\mathbf{A} \circ \mathbf{B} = \sum_{i=1}^{n}\sum_{j=1}^{n} \lambda_i\mu_j\,\bigl[(\mathbf{u}_i\mathbf{u}_i^{*}) \circ (\mathbf{w}_j\mathbf{w}_j^{*})\bigr] = \sum_{i,j} \lambda_i\mu_j\,\mathbf{z}_{ij}\mathbf{z}_{ij}^{*} ,$

where $\mathbf{z}_{ij} = \mathbf{u}_i \circ \mathbf{w}_j$ and the second equality is Lemma 13.7.1. Each $\mathbf{z}_{ij}\mathbf{z}_{ij}^{*}$ is positive semidefinite by Example (Five Positive Matrices) (b), and each coefficient $\lambda_i\mu_j$ is a non-negative real number. A non-negative combination of positive semidefinite matrices is positive semidefinite (Exercise (C2: The cone of positive matrices) (a)): the combination is Hermitian, being a real combination of Hermitian matrices, and for every $\mathbf{x}$ , $\langle \Bigl(\textstyle\sum_{i,j}\lambda_i\mu_j\mathbf{z}_{ij}\mathbf{z}_{ij}^{*}\Bigr)\mathbf{x}, \mathbf{x} \rangle = \sum_{i,j}\lambda_i\mu_j\langle \mathbf{z}_{ij}\mathbf{z}_{ij}^{*}\mathbf{x}, \mathbf{x} \rangle \ \ge\ 0 ,$ every term being a product of non-negative reals. So $\mathbf{A} \circ \mathbf{B} \succeq 0$ .
(b) Let $\varepsilon > 0$ be the smallest eigenvalue of $\mathbf{B}$ , which is positive by Theorem 13.1.5 (b). The matrix $\mathbf{B} - \varepsilon\mathbf{I}$ is Hermitian with eigenvalues $\mu_j - \varepsilon \ge 0$ , so $\mathbf{B} - \varepsilon\mathbf{I} \succeq 0$ by Theorem 13.1.4 (b). Since $\circ$ is bilinear and $\mathbf{A} \circ \mathbf{I} = \operatorname{diag}(a_{11}, \dots, a_{nn})$ ,

$\mathbf{A} \circ \mathbf{B} = \mathbf{A} \circ (\mathbf{B} - \varepsilon\mathbf{I}) \;+\; \varepsilon\,\operatorname{diag}(a_{11}, \dots, a_{nn}) .$

The first term is positive semidefinite by (a). In the second, $a_{ii} = \langle \mathbf{A}\mathbf{e}_i, \mathbf{e}_i \rangle > 0$ because $\mathbf{A} \succ 0$ and $\mathbf{e}_i \ne \mathbf{0}$ , so $\varepsilon\operatorname{diag}(a_{11}, \dots, a_{nn}) \succ 0$ by Example (Five Positive Matrices) (a). For $\mathbf{x} \ne \mathbf{0}$ , the quadratic form of the sum is $\ge 0$ plus $> 0$ , hence $> 0$ . So $\mathbf{A} \circ \mathbf{B} \succ 0$ , and the theorem is proved.

The theorem is a genuine addition to the list of operations that preserve positivity. Sums and non-negative multiples were easy (Exercise (C2: The cone of positive matrices)), congruence was easy (Proposition 13.1.2), and the ordinary product is not on the list at all, since $\mathbf{A}\mathbf{B}$ need not even be Hermitian. The entrywise product is on the list, and nothing in the definition of positivity suggests it should be.

Example (A

3 \times 3

Entrywise Product)

Let

$\mathbf{A} = \begin{pmatrix} 2 & 1 & 0 \\ 1 & 2 & 1 \\ 0 & 1 & 2 \end{pmatrix}, \qquad \mathbf{B} = \begin{pmatrix} 1 & -1 & 1 \\ -1 & 2 & -1 \\ 1 & -1 & 3 \end{pmatrix} .$

Check that $\mathbf{A} \succ 0$ and $\mathbf{B} \succ 0$ , compute $\mathbf{A} \circ \mathbf{B}$ , and verify directly that it is positive definite.

Solution. The leading principal minors of $\mathbf{A}$ are $2$ , $4 - 1 = 3$ and, expanding along the first row, $2(4-1) - 1(2-0) = 4$ . All are positive, so $\mathbf{A} \succ 0$ by Sylvester’s criterion (Theorem 13.1.5 (d)).

Those of $\mathbf{B}$ are $1$ , $2 - 1 = 1$ and $1(6-1) + 1(-3+1) + 1(1-2) = 5 - 2 - 1 = 2$ . All positive, so $\mathbf{B} \succ 0$ .

Entry by entry,

$\mathbf{A} \circ \mathbf{B} = \begin{pmatrix} 2 & -1 & 0 \\ -1 & 4 & -1 \\ 0 & -1 & 6 \end{pmatrix} ,$

whose leading principal minors are $2$ , $8 - 1 = 7$ and $2(24 - 1) + 1(-6 - 0) = 46 - 6 = 40$ . All positive, so $\mathbf{A} \circ \mathbf{B} \succ 0$ , as Theorem 13.7.2 (b) promised. Note that the eigenvalues of $\mathbf{A} \circ \mathbf{B}$ have nothing to do with those of $\mathbf{A}$ and $\mathbf{B}$ , and that $\det(\mathbf{A} \circ \mathbf{B}) = 40$ is much larger than $\det \mathbf{A}\det \mathbf{B} = 4 \cdot 2 = 8$ . The next theorem explains part of that: it forces $\det(\mathbf{A} \circ \mathbf{B}) \ge \det\mathbf{A} \prod_i b_{ii} = 24$ . Getting down to the product $\det\mathbf{A}\det\mathbf{B}$ needs Hadamard’s inequality (Theorem 13.6.4) as well, which is Exercise (C2: The product of the determinants).

Quick check. Let $\mathbf{A} = \begin{psmallmatrix} 1 & 1 \\ 1 & 1\end{psmallmatrix}$ and $\mathbf{B} = \begin{psmallmatrix} 2 & -1 \\ -1 & 2\end{psmallmatrix}$ . Compute $\mathbf{A} \circ \mathbf{B}$ and decide whether it is positive definite. Does this contradict Theorem 13.7.2 (b)?

Solution. $\mathbf{A} \circ \mathbf{B} = \begin{psmallmatrix} 2 & -1 \\ -1 & 2\end{psmallmatrix}$ , with leading principal minors $2$ and $3$ , so it is positive definite. There is no contradiction: Theorem 13.7.2 (b) is an implication, not an equivalence. Here $\mathbf{A}$ is only positive semidefinite ( $\mathbf{A} = \mathbf{1}\mathbf{1}^{\top}$ , so $\det \mathbf{A} = 0$ ), and the theorem’s hypothesis fails — but the conclusion happens to hold anyway. Part (a) is what applies, and it correctly predicts $\mathbf{A} \circ \mathbf{B} \succeq 0$ .

13.7.3. Oppenheim’s inequality

Hadamard’s inequality of Section 6 compared $\det \mathbf{A}$ with the product of the diagonal entries of $\mathbf{A}$ . Oppenheim’s inequality does the same for a Hadamard product, and it contains Hadamard’s as the special case $\mathbf{B} = \mathbf{I}$ .

Theorem 13.7.3 (Oppenheim’s Inequality)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ with $\mathbf{A} \succeq 0$ and $\mathbf{B} \succeq 0$ . Then

$\det(\mathbf{A} \circ \mathbf{B}) \ \ge \ \det \mathbf{A} \cdot b_{11}b_{22}\cdots b_{nn} .$

Idea. Induct on $n$ , taking both matrices positive definite first. Split off the $(1,1)$ entry of each: the Schur complements $\mathbf{A}/\alpha$ and $\mathbf{B}/\beta$ are positive definite of size $n - 1$ , and a short computation, whose only real step is Lemma 13.7.1 again, shows that the Schur complement of the corner in $\mathbf{A} \circ \mathbf{B}$ is

$(\mathbf{A} \circ \mathbf{B})/(\alpha\beta) = (\mathbf{A}/\alpha) \circ \mathbf{B}' + \tfrac1\alpha\,(\mathbf{a}\mathbf{a}^{*}) \circ (\mathbf{B}/\beta) ,$

a positive definite matrix plus a positive semidefinite one. So its determinant is at least that of the first term, by Lemma 13.6.2, and the inductive hypothesis handles that. The semidefinite case follows by replacing $\mathbf{A}$ and $\mathbf{B}$ with $\mathbf{A} + t\mathbf{I}$ and $\mathbf{B} + t\mathbf{I}$ and letting $t$ shrink to $0$ .

Proof. Case 1: $\mathbf{A} \succ 0$ and $\mathbf{B} \succ 0$ . Induct on $n$ . For $n = 1$ both sides are $a_{11}b_{11}$ .

Let $n \ge 2$ , and split both matrices with a $1 \times (n-1)$ partition:

$\mathbf{A} = \begin{pmatrix} \alpha & \mathbf{a}^{*} \\ \mathbf{a} & \mathbf{A}' \end{pmatrix}, \qquad \mathbf{B} = \begin{pmatrix} \beta & \mathbf{b}^{*} \\ \mathbf{b} & \mathbf{B}' \end{pmatrix},$

with $\alpha = a_{11} > 0$ , $\beta = b_{11} > 0$ and $\mathbf{a}, \mathbf{b} \in F^{n-1}$ . The Schur complements are

$\mathbf{A}/\alpha = \mathbf{A}' - \tfrac1\alpha\,\mathbf{a}\mathbf{a}^{*}, \qquad \mathbf{B}/\beta = \mathbf{B}' - \tfrac1\beta\,\mathbf{b}\mathbf{b}^{*} ,$

and both are positive definite by Theorem 13.6.1 (b). Since conjugation acts entrywise, $\mathbf{a}^{*} \circ \mathbf{b}^{*} = (\mathbf{a} \circ \mathbf{b})^{*}$ , so

$\mathbf{A} \circ \mathbf{B} = \begin{pmatrix} \alpha\beta & (\mathbf{a} \circ \mathbf{b})^{*} \\ \mathbf{a} \circ \mathbf{b} & \mathbf{A}' \circ \mathbf{B}' \end{pmatrix} ,$

and $\alpha\beta > 0$ , so this matrix too has a Schur complement at its corner:

$(\mathbf{A} \circ \mathbf{B})/(\alpha\beta) = \mathbf{A}' \circ \mathbf{B}' - \tfrac1{\alpha\beta}\,(\mathbf{a} \circ \mathbf{b})(\mathbf{a} \circ \mathbf{b})^{*} .$

By Lemma 13.7.1 the subtracted term is $\tfrac1{\alpha\beta}(\mathbf{a}\mathbf{a}^{*}) \circ (\mathbf{b}\mathbf{b}^{*})$ . Substituting $\mathbf{A}' = \mathbf{A}/\alpha + \tfrac1\alpha\mathbf{a}\mathbf{a}^{*}$ and expanding by bilinearity,

$\begin{aligned} \mathbf{A}' \circ \mathbf{B}' &= (\mathbf{A}/\alpha) \circ \mathbf{B}' + \tfrac1\alpha\,(\mathbf{a}\mathbf{a}^{*}) \circ \mathbf{B}' \\ &= (\mathbf{A}/\alpha) \circ \mathbf{B}' + \tfrac1\alpha\,(\mathbf{a}\mathbf{a}^{*}) \circ \Bigl(\mathbf{B}/\beta + \tfrac1\beta\mathbf{b}\mathbf{b}^{*}\Bigr) , \end{aligned}$

and the last term of this is exactly $\tfrac1{\alpha\beta}(\mathbf{a}\mathbf{a}^{*}) \circ (\mathbf{b}\mathbf{b}^{*})$ . It therefore cancels, leaving

$(\mathbf{A} \circ \mathbf{B})/(\alpha\beta) = \underbrace{(\mathbf{A}/\alpha) \circ \mathbf{B}'}_{\textstyle \succ 0} \;+\; \underbrace{\tfrac1\alpha\,(\mathbf{a}\mathbf{a}^{*}) \circ (\mathbf{B}/\beta)}_{\textstyle \succeq 0} . \tag{$\ast$}$

The first term is positive definite by Theorem 13.7.2 (b), since $\mathbf{A}/\alpha \succ 0$ and $\mathbf{B}' \succ 0$ (the latter by Theorem 13.6.1 (a) applied to $\mathbf{B}$ ). The second is positive semidefinite by Theorem 13.7.2 (a), since $\mathbf{a}\mathbf{a}^{*} \succeq 0$ by Example (Five Positive Matrices) (b) and $\mathbf{B}/\beta \succ 0$ , and $1/\alpha > 0$ .

Now compute. By Theorem 8.3.2 applied to $\mathbf{A} \circ \mathbf{B}$ , then $(\ast)$ with Lemma 13.6.2, then the inductive hypothesis applied to the pair $\mathbf{A}/\alpha$ and $\mathbf{B}'$ :

$\begin{aligned} \det(\mathbf{A} \circ \mathbf{B}) &= \alpha\beta\cdot\det\bigl[(\mathbf{A} \circ \mathbf{B})/(\alpha\beta)\bigr] \\ &\ge \alpha\beta\cdot\det\bigl[(\mathbf{A}/\alpha) \circ \mathbf{B}'\bigr] \\ &\ge \alpha\beta\cdot\det(\mathbf{A}/\alpha)\cdot b_{22}\cdots b_{nn} . \end{aligned}$

By Theorem 8.3.2 again, now for $\mathbf{A}$ , we have $\det(\mathbf{A}/\alpha) = \det \mathbf{A}/\alpha$ . Substituting and canceling $\alpha$ ,

$\det(\mathbf{A} \circ \mathbf{B}) \ \ge \ \beta\,\det \mathbf{A}\cdot b_{22}\cdots b_{nn} = \det \mathbf{A}\cdot b_{11}b_{22}\cdots b_{nn} ,$

which completes the induction.

Case 2: $\mathbf{A} \succeq 0$ and $\mathbf{B} \succeq 0$ . For real $t > 0$ , the matrices $\mathbf{A} + t\mathbf{I}$ and $\mathbf{B} + t\mathbf{I}$ are positive definite, as in the proof of Corollary 13.6.3. Case 1 gives

$\det\bigl[(\mathbf{A} + t\mathbf{I}) \circ (\mathbf{B} + t\mathbf{I})\bigr] \ \ge \ \det(\mathbf{A} + t\mathbf{I})\cdot\prod_{i=1}^{n}(b_{ii} + t) .$

Every entry of $(\mathbf{A} + t\mathbf{I}) \circ (\mathbf{B} + t\mathbf{I})$ is a polynomial in $t$ , so both sides are polynomial, hence continuous, functions of $t$ (Theorem 7.4.3). Letting $t$ decrease to $0$ preserves the inequality and gives the statement. This proves the theorem.

Remark. Taking $\mathbf{B} = \mathbf{I}$ recovers Hadamard’s inequality. Indeed $\mathbf{A} \circ \mathbf{I} = \operatorname{diag}(a_{11}, \dots, a_{nn})$ and $\prod_i b_{ii} = 1$ , so the conclusion reads $a_{11}\cdots a_{nn} \ge \det \mathbf{A}$ , which is Theorem 13.6.4 (a). Swapping the roles of $\mathbf{A}$ and $\mathbf{B}$ gives the companion bound $\det(\mathbf{A} \circ \mathbf{B}) \ge \det \mathbf{B}\cdot\prod_i a_{ii}$ , and combining either one with Hadamard’s inequality gives $\det(\mathbf{A} \circ \mathbf{B}) \ge \det \mathbf{A}\det \mathbf{B}$ ; that last consequence is Exercise (C2: The product of the determinants).

13.7.4. Where this gets used

The natural home of the Hadamard product is data. A covariance matrix is a Gram matrix (Theorem 11.2.5), hence positive semidefinite, and rescaling each variable to unit variance turns it into a correlation matrix: positive semidefinite with every diagonal entry equal to $1$ . The Schur product theorem says that the entrywise product of two correlation matrices is another one — the diagonal entries stay $1 \cdot 1 = 1$ — so one can damp the correlations of one model by those of another and still have a legitimate covariance structure. Nothing in the definition of “positive semidefinite” makes that obvious, and it is the reason the theorem is quoted far outside linear algebra.

Warning. Entrywise products preserve positivity; entrywise inverses do not. Let $\mathbf{A} = \begin{psmallmatrix} 2 & 1 \\ 1 & 2\end{psmallmatrix}$ , which is positive definite, with leading principal minors $2$ and $3$ . Replacing each entry by its reciprocal gives $\begin{psmallmatrix} 1/2 & 1 \\ 1 & 1/2\end{psmallmatrix}$ , whose determinant is $\tfrac14 - 1 = -\tfrac34 < 0$ : not positive semidefinite. The entrywise reciprocal is not the Hadamard inverse of anything useful, and the matrix inverse $\mathbf{A}^{-1}$ , which is positive definite (Exercise (B3: The inverse of a positive definite matrix)), has nothing to do with it.

13.7.5. Exercises

A. Check your understanding

Exercise (A1)

(a) Define $\mathbf{A} \circ \mathbf{B}$ , and say what shapes the two factors must have.
(b) What is $\mathbf{A} \circ \mathbf{I}$ ? What is $\mathbf{A} \circ \mathbf{J}$ ?
(c) State the Schur product theorem.
(d) True or false: if $\mathbf{A} \succ 0$ , then the matrix with entries $1/a_{ij}$ (assuming no entry is $0$ ) is positive definite. Justify your answer.
(e) What does Oppenheim’s inequality say when $\mathbf{B} = \mathbf{I}$ ?

Solution.

(a) For $\mathbf{A}, \mathbf{B} \in M_{m\times n}(F)$ of the same shape, $(\mathbf{A} \circ \mathbf{B})_{ij} = a_{ij}b_{ij}$ , and $\mathbf{A} \circ \mathbf{B}$ has that same shape (Definition 13.7.1).
(b) $\mathbf{A} \circ \mathbf{I} = \operatorname{diag}(a_{11}, \dots, a_{nn})$ , and $\mathbf{A} \circ \mathbf{J} = \mathbf{A}$ , since $\mathbf{J}$ has every entry $1$ .
(c) If $\mathbf{A} \succeq 0$ and $\mathbf{B} \succeq 0$ , then $\mathbf{A} \circ \mathbf{B} \succeq 0$ ; if both are positive definite, so is $\mathbf{A} \circ \mathbf{B}$ (Theorem 13.7.2).
(d) False. For $\mathbf{A} = \begin{psmallmatrix} 2 & 1 \\ 1 & 2\end{psmallmatrix} \succ 0$ , the entrywise reciprocal is $\begin{psmallmatrix} 1/2 & 1 \\ 1 & 1/2\end{psmallmatrix}$ , with determinant $-3/4 < 0$ .
(e) By Theorem 13.7.3 it says $\det(\mathbf{A} \circ \mathbf{I}) \ge \det \mathbf{A} \cdot 1$ , that is $a_{11}\cdots a_{nn} \ge \det \mathbf{A}$ : Hadamard’s inequality, Theorem 13.6.4 (a).

B. Practice

Exercise (B1: Compute and decide)

For each pair below, compute $\mathbf{A} \circ \mathbf{B}$ and determine whether it is positive definite, positive semidefinite but not definite, or neither. Say in each case whether Theorem 13.7.2 applies.

(a) $\mathbf{A} = \begin{psmallmatrix} 1 & 1 \\ 1 & 1\end{psmallmatrix}$ , $\mathbf{B} = \begin{psmallmatrix} 1 & 1 \\ 1 & 1\end{psmallmatrix}$ .
(b) $\mathbf{A} = \begin{psmallmatrix} 1 & 1 \\ 1 & 1\end{psmallmatrix}$ , $\mathbf{B} = \begin{psmallmatrix} 1 & 2 \\ 2 & 1\end{psmallmatrix}$ .
(c) $\mathbf{A} = \begin{psmallmatrix} 2 & 1 \\ 1 & 2\end{psmallmatrix}$ , $\mathbf{B} = \begin{psmallmatrix} 3 & -1 \\ -1 & 1\end{psmallmatrix}$ .

Solution.

(a) $\mathbf{A} \circ \mathbf{B} = \begin{psmallmatrix} 1 & 1 \\ 1 & 1\end{psmallmatrix}$ . Both factors are $\mathbf{1}\mathbf{1}^{\top} \succeq 0$ (Example (Five Positive Matrices) (b)), so Theorem 13.7.2 (a) applies and predicts $\succeq 0$ . The product has determinant $0$ , so it is positive semidefinite but not definite.
(b) $\mathbf{A} \circ \mathbf{B} = \begin{psmallmatrix} 1 & 2 \\ 2 & 1\end{psmallmatrix}$ , whose determinant is $1 - 4 = -3 < 0$ : neither. The theorem does not apply, since $\mathbf{B}$ is not positive semidefinite (its determinant is $-3$ ). This is what the hypothesis is for.
(c) $\mathbf{A} \succ 0$ (minors $2$ , $3$ ) and $\mathbf{B} \succ 0$ (minors $3$ , $2$ ), so Theorem 13.7.2 (b) applies. Indeed $\mathbf{A} \circ \mathbf{B} = \begin{psmallmatrix} 6 & -1 \\ -1 & 2\end{psmallmatrix}$ , with minors $6$ and $11$ , is positive definite. Theorem 13.7.3 also checks out: $11 \ge \det \mathbf{A}\cdot b_{11}b_{22} = 3 \cdot 3 = 9$ .

Exercise (B2: Entrywise squares)

Let $\mathbf{A} = \begin{psmallmatrix} 1 & 1 & 1 \\ 1 & 2 & 2 \\ 1 & 2 & 3\end{psmallmatrix}$ .

(a) Show that $\mathbf{A} \succ 0$ .
(b) Compute $\mathbf{A} \circ \mathbf{A}$ , the matrix of squared entries, and verify that it is positive definite.
(c) Explain why (b) needed no computation.

Solution.

(a) The leading principal minors are $1$ , $2 - 1 = 1$ and, expanding along the first row, $1(6-4) - 1(3-2) + 1(2-2) = 1$ . All positive, so $\mathbf{A} \succ 0$ by Theorem 13.1.5 (d).
(b) $\mathbf{A} \circ \mathbf{A} = \begin{psmallmatrix} 1 & 1 & 1 \\ 1 & 4 & 4 \\ 1 & 4 & 9\end{psmallmatrix}$ , with leading principal minors $1$ , $4 - 1 = 3$ and $1(36 - 16) - 1(9 - 4) + 1(4 - 4) = 20 - 5 = 15$ . All positive, so it is positive definite.
(c) Theorem 13.7.2 (b) with $\mathbf{B} = \mathbf{A}$ gives $\mathbf{A} \circ \mathbf{A} \succ 0$ immediately.

C. Going deeper

Exercise (C1: Polynomials applied entrywise)

Let $\mathbf{A} \in M_n(F)$ with $\mathbf{A} \succeq 0$ .

(a) Prove that the entrywise power $\mathbf{A}^{\circ m}$ , with $(i, j)$ entry $a_{ij}^{\,m}$ , is positive semidefinite for every integer $m \ge 1$ .
(b) Let $p(x) = c_0 + c_1x + \dots + c_dx^d$ have non-negative real coefficients. Prove that the matrix $p^{\circ}(\mathbf{A})$ with $(i, j)$ entry $p(a_{ij})$ is positive semidefinite.
(c) Give an example of $\mathbf{A} \succeq 0$ and a polynomial $p$ with a negative coefficient for which $p^{\circ}(\mathbf{A})$ is not positive semidefinite.

Solution.

(a) Induct on $m$ . For $m = 1$ the claim is the hypothesis. If $\mathbf{A}^{\circ m} \succeq 0$ , then $\mathbf{A}^{\circ(m+1)} = \mathbf{A}^{\circ m} \circ \mathbf{A} \succeq 0$ by Theorem 13.7.2 (a).
(b) By definition of the entries, $p^{\circ}(\mathbf{A}) = c_0\mathbf{J} + c_1\mathbf{A} + c_2\mathbf{A}^{\circ 2} + \dots + c_d\mathbf{A}^{\circ d}$ , since each side has $(i, j)$ entry $\sum_k c_ka_{ij}^{\,k}$ (with $a_{ij}^0 = 1$ , which is the $(i,j)$ entry of $\mathbf{J}$ ). Now $\mathbf{J} = \mathbf{1}\mathbf{1}^{\top} \succeq 0$ by Example (Five Positive Matrices) (b), each $\mathbf{A}^{\circ k} \succeq 0$ by (a), and each $c_k \ge 0$ . A non-negative combination of positive semidefinite matrices is positive semidefinite (Exercise (C2: The cone of positive matrices) (a)).
(c) Take $\mathbf{A} = \mathbf{J} \in M_2(\mathbb{R})$ , which is positive semidefinite, and $p(x) = x - 2$ . Then $p^{\circ}(\mathbf{A})$ has every entry $-1$ , that is $p^{\circ}(\mathbf{A}) = -\mathbf{J}$ , and $\langle -\mathbf{J}\mathbf{e}_1, \mathbf{e}_1 \rangle = -1 < 0$ . So the hypothesis on the coefficients cannot be dropped.

Exercise (C2: The product of the determinants)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ with $\mathbf{A} \succeq 0$ and $\mathbf{B} \succeq 0$ . Prove that

$\det(\mathbf{A} \circ \mathbf{B}) \ \ge \ \det \mathbf{A} \cdot \det \mathbf{B} .$

Is the inequality ever an equality with both matrices positive definite?

Solution. By Theorem 13.7.3, $\det(\mathbf{A} \circ \mathbf{B}) \ge \det \mathbf{A}\cdot\prod_i b_{ii}$ . By Theorem 13.6.4 (a) applied to $\mathbf{B} \succeq 0$ , $\prod_i b_{ii} \ge \det \mathbf{B}$ . Both $\det \mathbf{A} \ge 0$ and $\prod_i b_{ii} \ge 0$ (the diagonal entries of a positive semidefinite matrix are $\ge 0$ by Exercise (C1: Zero trace forces zero) (a)), so multiplying the second inequality by $\det \mathbf{A} \ge 0$ and chaining,

$\det(\mathbf{A} \circ \mathbf{B}) \ \ge\ \det \mathbf{A}\prod_i b_{ii} \ \ge\ \det \mathbf{A}\det \mathbf{B} .$

Yes, equality can occur: take $\mathbf{A} = \mathbf{B} = \mathbf{I}_n$ , so that $\mathbf{A} \circ \mathbf{B} = \mathbf{I}_n$ and all three quantities are $1$ . More generally, take $\mathbf{A} \succ 0$ diagonal and $\mathbf{B} = \mathbf{I}_n$ .

Exercise (C3: The rank of an entrywise product)

Let $\mathbf{A}, \mathbf{B} \in M_n(F)$ with $\mathbf{A} \succeq 0$ and $\mathbf{B} \succeq 0$ , of ranks $r$ and $s$ . Prove that

$\operatorname{rank}(\mathbf{A} \circ \mathbf{B}) \le rs .$

Hint: how many terms in the proof of Theorem 13.7.2 (a) are actually non-zero?

Solution. Write $\mathbf{A} = \sum_i\lambda_i\mathbf{u}_i\mathbf{u}_i^{*}$ and $\mathbf{B} = \sum_j\mu_j\mathbf{w}_j\mathbf{w}_j^{*}$ as in the proof of Theorem 13.7.2 (a). The number of non-zero eigenvalues of $\mathbf{A}$ is $\operatorname{rank} \mathbf{A} = r$ : diagonalizing, $\mathbf{A} = \mathbf{U}\mathbf{D}\mathbf{U}^{*}$ with $\mathbf{U}$ invertible, so $\operatorname{rank} \mathbf{A} = \operatorname{rank} \mathbf{D}$ by Theorem 3.5.9, and $\operatorname{rank} \mathbf{D}$ counts the non-zero $\lambda_i$ . Likewise exactly $s$ of the $\mu_j$ are non-zero. Dropping the vanishing terms,

$\mathbf{A} \circ \mathbf{B} = \sum_{\lambda_i \ne 0}\ \sum_{\mu_j \ne 0} \lambda_i\mu_j\,\mathbf{z}_{ij}\mathbf{z}_{ij}^{*}, \qquad \mathbf{z}_{ij} = \mathbf{u}_i \circ \mathbf{w}_j ,$

a sum of $rs$ matrices, each of rank at most $1$ , since every column of $\mathbf{z}_{ij}\mathbf{z}_{ij}^{*}$ lies in $\operatorname{span}(\mathbf{z}_{ij})$ . The rank of a sum of two matrices is at most the sum of their ranks (Exercise (C1: Rank of a sum)); applying that repeatedly to the $rs$ summands gives $\operatorname{rank}(\mathbf{A} \circ \mathbf{B}) \le rs$ .