11.1 Inner Products and Norms

Nine chapters, and we have not measured anything. We can say that a list is independent, that a subspace has dimension four, that an operator is diagonalizable. We cannot yet say that a vector has length $3$ , or that two vectors meet at a right angle. Geometry has appeared only in pictures, never in the theory. This section puts it into the theory, by abstracting the one operation in $\mathbb{R}^n$ that already computes lengths and angles: the dot product.

11.1.1. The field for this chapter

Throughout Chapter 11 the field $F$ is $\mathbb{R}$ or $\mathbb{C}$ . This is not laziness; two features of these fields are used on every page, and no other field has both.

The first is positivity. The point of the whole construction is that $\langle \mathbf{v}, \mathbf{v} \rangle$ should be the square of a length, so it must be a number we can compare with $0$ . Over $\mathbb{F}_p$ there is no ordering compatible with the arithmetic at all: in $\mathbb{F}_5$ , adding $1$ to itself five times returns to $0$ , so no non-zero element can be called “positive” in any useful sense. Over $\mathbb{Q}$ there is an ordering, but lengths would not exist, since $\sqrt{2} \notin \mathbb{Q}$ . Over $\mathbb{R}$ every sum of squares is $\ge 0$ and every non-negative number has a square root, and that is exactly what we need.

The second is conjugation. Over $\mathbb{C}$ the naive sum of squares fails immediately: $1^2 + i^2 = 0$ , so “ $z_1^2 + z_2^2$ ” is not a length for $(1, i) \ne \mathbf{0}$ . The repair is to use $z\overline{z} = \lvert z \rvert^2$ instead of $z^2$ (Theorem 0.6.2 (c)), and this needs the complex conjugate. We write $\overline{a}$ for the conjugate of $a \in F$ , with the convention that $\overline{a} = a$ when $F = \mathbb{R}$ ; every formula below then reads correctly over both fields at once.

11.1.2. From the dot product to the axioms

In $\mathbb{R}^n$ the dot product is $\mathbf{x} \cdot \mathbf{y} = x_1y_1 + \dots + x_ny_n .$ It computes lengths, because $\mathbf{x} \cdot \mathbf{x} = x_1^2 + \dots + x_n^2$ is the square of the Euclidean length by Pythagoras. It computes angles too. For non-zero $\mathbf{x}, \mathbf{y} \in \mathbb{R}^2$ at an angle $\theta$ , the law of cosines applied to the triangle with sides $\mathbf{x}$ , $\mathbf{y}$ and $\mathbf{x} - \mathbf{y}$ says $\lVert \mathbf{x} - \mathbf{y} \rVert^2 = \lVert \mathbf{x} \rVert^2 + \lVert \mathbf{y} \rVert^2 - 2\lVert \mathbf{x} \rVert \lVert \mathbf{y} \rVert \cos\theta ,$ while expanding the left side entry by entry gives $\lVert \mathbf{x} \rVert^2 + \lVert \mathbf{y} \rVert^2 - 2(\mathbf{x} \cdot \mathbf{y})$ . Comparing the two, $\mathbf{x} \cdot \mathbf{y} = \lVert \mathbf{x} \rVert \lVert \mathbf{y} \rVert \cos\theta$ . So one bilinear expression carries both length and angle, and perpendicularity is the single equation $\mathbf{x} \cdot \mathbf{y} = 0$ .

Which properties of the dot product did that argument use? Only three: it is linear in each slot separately, it is symmetric, and $\mathbf{x} \cdot \mathbf{x} > 0$ for $\mathbf{x} \ne \mathbf{0}$ . Those three we can demand of an abstract space. Over $\mathbb{C}$ , as we saw, symmetry has to be weakened to conjugate symmetry, and the price is small: a conjugate bar in the second slot.

An inner product is a way of multiplying two vectors into a scalar so that a vector paired with itself has positive “length squared”.

Definition 11.1.1 (Inner Product)

Let $F$ be $\mathbb{R}$ or $\mathbb{C}$ , and let $V$ be a vector space over $F$ . An inner product on $V$ is a map $\langle \cdot, \cdot \rangle \colon V \times V \to F$ such that for all $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and all $a, b \in F$ :

(IP1) $\langle a\mathbf{u} + b\mathbf{v}, \mathbf{w} \rangle = a\langle \mathbf{u}, \mathbf{w} \rangle + b\langle \mathbf{v}, \mathbf{w} \rangle$ (linear in the first slot);
(IP2) $\langle \mathbf{v}, \mathbf{u} \rangle = \overline{\langle \mathbf{u}, \mathbf{v} \rangle}$ (conjugate symmetry);
(IP3) $\langle \mathbf{v}, \mathbf{v} \rangle > 0$ for every non-zero $\mathbf{v} \in V$ (positive definiteness).

A vector space equipped with an inner product is an inner product space. When $F = \mathbb{R}$ it is a real inner product space, when $F = \mathbb{C}$ a complex one.

In words, clause by clause. (IP1) says that if we freeze the second slot, what is left is a linear functional on $V$ : the inner product distributes over sums and lets scalars out of the first argument untouched. (IP2) says that swapping the two arguments conjugates the answer; over $\mathbb{R}$ it is plain symmetry, $\langle \mathbf{v}, \mathbf{u} \rangle = \langle \mathbf{u}, \mathbf{v} \rangle$ . (IP3) says that a non-zero vector pairs with itself to give a strictly positive number. For the inequality in (IP3) to mean anything, $\langle \mathbf{v}, \mathbf{v} \rangle$ must be a real number; it is, and the reason is (IP2), as the next result shows. Note also what (IP3) does not say: it says nothing about $\langle \mathbf{u}, \mathbf{v} \rangle$ for $\mathbf{u} \ne \mathbf{v}$ , which may be negative, or not even real.

The first consequences are small, and we will use all of them constantly.

Theorem 11.1.1 (First Properties of an Inner Product)

Let $V$ be an inner product space over $F$ . For all $\mathbf{u}, \mathbf{v}, \mathbf{w} \in V$ and $a, b \in F$ :

(a) $\langle \mathbf{v}, \mathbf{v} \rangle$ is a real number, and $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$ with equality if and only if $\mathbf{v} = \mathbf{0}$ ;
(b) $\langle \mathbf{u}, a\mathbf{v} + b\mathbf{w} \rangle = \overline{a}\langle \mathbf{u}, \mathbf{v} \rangle + \overline{b}\langle \mathbf{u}, \mathbf{w} \rangle$ (conjugate-linear in the second slot);
(c) $\langle \mathbf{0}, \mathbf{v} \rangle = \langle \mathbf{v}, \mathbf{0} \rangle = 0$ ;
(d) if $\langle \mathbf{u}, \mathbf{w} \rangle = \langle \mathbf{v}, \mathbf{w} \rangle$ for every $\mathbf{w} \in V$ , then $\mathbf{u} = \mathbf{v}$ .

Idea. Each part is one application of an axiom. For (a), a number equal to its own conjugate is real. For (b), swap slots with (IP2), use (IP1), and swap back. For (d), the only vector orthogonal to everything is the zero vector, and we find that out by pairing the difference with itself: that is the move to remember.

Proof.

(a) By (IP2) with $\mathbf{u} = \mathbf{v}$ , $\langle \mathbf{v}, \mathbf{v} \rangle = \overline{\langle \mathbf{v}, \mathbf{v} \rangle}$ , and a complex number equal to its own conjugate has zero imaginary part, so $\langle \mathbf{v}, \mathbf{v} \rangle \in \mathbb{R}$ . If $\mathbf{v} \ne \mathbf{0}$ , then $\langle \mathbf{v}, \mathbf{v} \rangle > 0$ by (IP3). If $\mathbf{v} = \mathbf{0}$ , then $\langle \mathbf{0}, \mathbf{0} \rangle = \langle 0 \cdot \mathbf{0}, \mathbf{0} \rangle = 0\langle \mathbf{0}, \mathbf{0} \rangle = 0$ by (IP1). Hence $\langle \mathbf{v}, \mathbf{v} \rangle \ge 0$ always, with equality exactly when $\mathbf{v} = \mathbf{0}$ .
(b) Using (IP2), then (IP1), then (IP2) again, $\begin{aligned} \langle \mathbf{u}, a\mathbf{v} + b\mathbf{w} \rangle &= \overline{\langle a\mathbf{v} + b\mathbf{w}, \mathbf{u} \rangle} \\ &= \overline{a\langle \mathbf{v}, \mathbf{u} \rangle + b\langle \mathbf{w}, \mathbf{u} \rangle} \\ &= \overline{a}\,\overline{\langle \mathbf{v}, \mathbf{u} \rangle} + \overline{b}\,\overline{\langle \mathbf{w}, \mathbf{u} \rangle} \\ &= \overline{a}\langle \mathbf{u}, \mathbf{v} \rangle + \overline{b}\langle \mathbf{u}, \mathbf{w} \rangle, \end{aligned}$ where the third equality uses Theorem 0.6.2 (a) and (b).
(c) $\langle \mathbf{0}, \mathbf{v} \rangle = \langle 0 \cdot \mathbf{0}, \mathbf{v} \rangle = 0\langle \mathbf{0}, \mathbf{v} \rangle = 0$ by (IP1), and $\langle \mathbf{v}, \mathbf{0} \rangle = \overline{\langle \mathbf{0}, \mathbf{v} \rangle} = 0$ by (IP2).
(d) Put $\mathbf{z} = \mathbf{u} - \mathbf{v}$ . For every $\mathbf{w}$ , (IP1) gives $\langle \mathbf{z}, \mathbf{w} \rangle = \langle \mathbf{u}, \mathbf{w} \rangle - \langle \mathbf{v}, \mathbf{w} \rangle = 0$ . Take $\mathbf{w} = \mathbf{z}$ : then $\langle \mathbf{z}, \mathbf{z} \rangle = 0$ , so $\mathbf{z} = \mathbf{0}$ by (a), that is, $\mathbf{u} = \mathbf{v}$ .

Part (d) is the first appearance of the move that runs through the whole chapter: to show a vector is zero, pair it with itself. There is exactly one vector of length zero, so an identity of the form $\langle \mathbf{z}, \mathbf{z} \rangle = 0$ is a proof that $\mathbf{z} = \mathbf{0}$ .

Part (b) is where the conjugate becomes visible in practice. An inner product over $\mathbb{C}$ is not bilinear: scalars leave the second slot wearing a bar. A map with properties (IP1) and (b) is called sesquilinear, from the Latin for “one and a half times linear”.

11.1.3. Examples

The dot product on $\mathbb{R}^n$ . Define $\langle \mathbf{x}, \mathbf{y} \rangle = x_1y_1 + \dots + x_ny_n$ . (IP1) holds because each term is linear in $x_i$ ; (IP2) is the commutativity of multiplication in $\mathbb{R}$ , and conjugation is the identity; (IP3) holds because $\langle \mathbf{x}, \mathbf{x} \rangle = x_1^2 + \dots + x_n^2$ is a sum of squares of real numbers, which is $0$ only when every $x_i = 0$ . This is the standard inner product on $\mathbb{R}^n$ , and it is the model for everything else.

The standard inner product on $\mathbb{C}^n$ . Define $\langle \mathbf{x}, \mathbf{y} \rangle = x_1\overline{y_1} + \dots + x_n\overline{y_n} .$ (IP1) is clear. For (IP2), $\overline{\langle \mathbf{x}, \mathbf{y} \rangle} = \sum \overline{x_i}y_i = \langle \mathbf{y}, \mathbf{x} \rangle$ by Theorem 0.6.2 (a), (b). For (IP3), $\langle \mathbf{x}, \mathbf{x} \rangle = \sum x_i\overline{x_i} = \sum \lvert x_i \rvert^2$ by Theorem 0.6.2 (c), a sum of non-negative real numbers, zero only when every $x_i = 0$ . The conjugates are doing exactly one job: making the diagonal values real and non-negative.

Weighted inner products. Fix real numbers $w_1, \dots, w_n > 0$ and put $\langle \mathbf{x}, \mathbf{y} \rangle = \sum_i w_i x_i\overline{y_i}$ on $F^n$ . The same three checks go through, with $\langle \mathbf{x}, \mathbf{x} \rangle = \sum w_i\lvert x_i \rvert^2$ ; the weights must be strictly positive, or (IP3) fails for a standard basis vector. In the degenerate case $n = 1$ and $w_1 = 1$ , this is $\langle x, y \rangle = x\overline{y}$ on $F$ itself, the smallest non-zero inner product space there is. It matters because it is where the axioms are just the arithmetic of $\lvert \cdot \rvert$ .

Continuous functions. Let $V = C[0, 1]$ be the space of continuous functions $f \colon [0, 1] \to F$ , and set $\langle f, g \rangle = \int_0^1 f(t)\overline{g(t)}\,\mathrm{d} t .$ (IP1) is linearity of the integral, and (IP2) follows because conjugation commutes with integration. For (IP3), $\langle f, f \rangle = \int_0^1 \lvert f(t) \rvert^2 \,\mathrm{d} t \ge 0$ , and we must rule out equality for $f \ne 0$ . Here is the one fact from analysis we assume: if $f$ is continuous and $f(c) \ne 0$ for some $c$ , then $\lvert f \rvert \ge \lvert f(c) \rvert / 2$ on some interval of positive length $\delta$ around $c$ , so $\int_0^1 \lvert f \rvert^2 \ge \delta \lvert f(c) \rvert^2 / 4 > 0$ . Continuity is not decoration; the non-example below shows what happens without it.

Matrices: the Frobenius inner product. First a piece of notation we will need all chapter.

Definition 11.1.2 (Conjugate Transpose)

For $\mathbf{A} \in M_{m \times n}(F)$ , the conjugate transpose $\mathbf{A}^{*} \in M_{n \times m}(F)$ is the matrix with entries $(\mathbf{A}^{*})_{ij} = \overline{a_{ji}}$ . A square matrix with $\mathbf{A}^{*} = \mathbf{A}$ is Hermitian.

Over $\mathbb{R}$ , $\mathbf{A}^{*} = \mathbf{A}^{\top}$ and Hermitian means symmetric (Definition 0.9.11). Now on $V = M_{m \times n}(F)$ define $\langle \mathbf{A}, \mathbf{B} \rangle = \operatorname{tr}(\mathbf{B}^{*}\mathbf{A}) .$ Writing out the trace, $\operatorname{tr}(\mathbf{B}^{*}\mathbf{A}) = \sum_{j}\sum_{i} \overline{b_{ij}}a_{ij}$ , so this is the standard inner product of $F^{mn}$ after listing the entries in a column. In particular (IP1)–(IP3) hold, with $\langle \mathbf{A}, \mathbf{A} \rangle = \sum_{i,j}\lvert a_{ij} \rvert^2$ . This is the Frobenius inner product; for $n = 1$ it is the standard inner product on $F^m$ .

Polynomials sampled at nodes. Fix $n + 1$ distinct scalars $c_0, \dots, c_n \in F$ and define, on $V = F[x]_{\le n}$ , $\langle p, q \rangle = \sum_{i=0}^{n} p(c_i)\overline{q(c_i)} .$ (IP1) and (IP2) hold because evaluation at $c_i$ is linear (Theorem 0.8.5) and the standard inner product on $F^{n+1}$ has them; indeed $\langle p, q \rangle$ is the standard inner product of the sample vectors $(p(c_0), \dots, p(c_n))$ and $(q(c_0), \dots, q(c_n))$ . Positivity is the interesting clause. If $\langle p, p \rangle = \sum \lvert p(c_i) \rvert^2 = 0$ , then $p$ vanishes at all $n + 1$ distinct nodes. A non-zero polynomial of degree at most $n$ has at most $n$ distinct roots (Corollary 6.1.4), so $p = 0$ . The number of nodes is what makes this work, and the non-example below removes one.

11.1.4. Non-examples, by minimal change

Change a sign: the hyperbolic form on $\mathbb{R}^2$ . Let $\beta(\mathbf{x}, \mathbf{y}) = x_1y_1 - x_2y_2$ . Bilinearity survives, since each term is still linear in each slot, and so does symmetry. What fails is (IP3), and the witness is $\mathbf{e}_2$ : $\beta(\mathbf{e}_2, \mathbf{e}_2) = -1 < 0$ . The vector $(1, 1)$ is worse still: it is non-zero, yet $\beta((1,1),(1,1)) = 0$ , so it would be a non-zero vector of length $0$ . Forms like $\beta$ are useful (they are the subject of the chapter on bilinear forms), but they are not inner products.

Drop the conjugate on $\mathbb{C}^2$ . Let $\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^{\top}\mathbf{v} = u_1v_1 + u_2v_2$ on $\mathbb{C}^2$ . This map is linear in both slots and symmetric, so it satisfies (IP1) and looks tidier than the standard product. Take $\mathbf{u} = (1, i)$ . Then $\langle \mathbf{u}, \mathbf{u} \rangle = 1^2 + i^2 = 1 - 1 = 0,$ with $\mathbf{u} \ne \mathbf{0}$ . Clause (IP3) fails. This one example is the whole reason for the conjugate: over $\mathbb{C}$ , sums of squares vanish on non-zero vectors, sums of squared moduli do not.

Enlarge the space: drop continuity. Keep the formula $\langle f, g \rangle = \int_0^1 fg$ , but let $V$ be all Riemann-integrable functions $[0, 1] \to \mathbb{R}$ . Linearity and symmetry are untouched. Let $f$ be $1$ at $t = \tfrac12$ and $0$ elsewhere. It is integrable, non-zero as a function, and $\int_0^1 f^2 = 0$ . Positivity fails, so $\langle \cdot, \cdot \rangle$ is only positive semidefinite here. There are two standard repairs: restrict to continuous functions, as above, or regard two functions as equal when they differ on a negligible set. We take the first, and every function space in this chapter will be a space of continuous functions.

Remove a node. On $\mathbb{R}[x]_{\le 2}$ , put $\langle p, q \rangle = p(0)q(0) + p(1)q(1)$ : the node example with one node too few. Bilinearity and symmetry hold. But $p = x^2 - x$ is a non-zero polynomial of degree $2$ with $p(0) = p(1) = 0$ , so $\langle p, p \rangle = 0$ . The clause that fails is (IP3), and it fails because two nodes cannot pin down a quadratic.

Why this definition. Each clause earns its place. Without (IP3) there can be non-zero vectors $\mathbf{v}$ with $\langle \mathbf{v}, \mathbf{v} \rangle = 0$ , and the proof of Theorem 11.1.1 (d) breaks down, because pairing a vector with itself no longer detects zero. (The form $x_1y_1 - x_2y_2$ below shows that the conclusion of (d) may survive anyway; what is lost is the argument for it, and with it every later proof that runs through $\langle \mathbf{v}, \mathbf{v} \rangle = 0 \Rightarrow \mathbf{v} = \mathbf{0}$ .) Without (IP2) the diagonal values need not be real, and “ $\langle \mathbf{v}, \mathbf{v} \rangle > 0$ ” is meaningless. Linearity in the first slot rather than the second is a convention, but once chosen it must be kept: with (IP2) in place, linearity in one slot forces conjugate-linearity in the other, so a map cannot be linear in both unless $F = \mathbb{R}$ .

Warning. The conjugate goes in the second slot here, and books differ. With our convention, $\langle a\mathbf{u}, \mathbf{v} \rangle = a\langle \mathbf{u}, \mathbf{v} \rangle$ and $\langle \mathbf{u}, a\mathbf{v} \rangle = \overline{a}\langle \mathbf{u}, \mathbf{v} \rangle$ . Many books, especially in physics, put the conjugate in the first slot, so that their $\langle \mathbf{u}, \mathbf{v} \rangle$ is our $\langle \mathbf{v}, \mathbf{u} \rangle$ . Every formula involving a lone conjugate then flips. When you read a formula elsewhere, check the convention first; when you write one, say which you use.

Quick check. Is $\langle \mathbf{x}, \mathbf{y} \rangle = x_1y_1 + x_2y_2 + x_1y_2$ an inner product on $\mathbb{R}^2$ ? If not, name the clause that fails and give a witness.

Solution. No. It is bilinear, and $\langle \mathbf{x}, \mathbf{x} \rangle = x_1^2 + x_2^2 + x_1x_2 = (x_1 + \tfrac12 x_2)^2 + \tfrac34 x_2^2 > 0$ for $\mathbf{x} \ne \mathbf{0}$ , so (IP1) and (IP3) both hold. The clause that fails is (IP2): with $\mathbf{u} = \mathbf{e}_1$ and $\mathbf{v} = \mathbf{e}_2$ we get $\langle \mathbf{u}, \mathbf{v} \rangle = 1$ but $\langle \mathbf{v}, \mathbf{u} \rangle = 0$ . Symmetrizing the cross term, to $x_1y_1 + x_2y_2 + \tfrac12(x_1y_2 + x_2y_1)$ , repairs it.

11.1.5. Norms

By Theorem 11.1.1 (a), the number $\langle \mathbf{v}, \mathbf{v} \rangle$ is real and $\ge 0$ , so it has a unique non-negative real square root. That square root is the length we were after.

Definition 11.1.3 (Induced Norm)

Let $V$ be an inner product space. The norm of $\mathbf{v} \in V$ is $\left\| \mathbf{v} \right\| \mathrel{:=} \sqrt{\langle \mathbf{v}, \mathbf{v} \rangle} ,$ the non-negative real square root. A vector with $\left\| \mathbf{v} \right\| = 1$ is a unit vector, and replacing a non-zero $\mathbf{v}$ by $\mathbf{v} / \left\| \mathbf{v} \right\|$ is called normalizing it.

On $\mathbb{R}^n$ with the dot product this is the Euclidean length $\sqrt{x_1^2 + \dots + x_n^2}$ , and on $\mathbb{C}^n$ it is $\sqrt{\sum \lvert x_i \rvert^2}$ . On $C[0,1]$ it is $\big(\int_0^1 \lvert f \rvert^2\big)^{1/2}$ , and on matrices it is the Frobenius norm $\big(\sum \lvert a_{ij} \rvert^2\big)^{1/2}$ . By definition $\left\| \mathbf{v} \right\|^2 = \langle \mathbf{v}, \mathbf{v} \rangle$ always, and this identity, read from right to left, is the most-used step in the chapter: work with the norm squared, then expand.

Theorem 11.1.2 (Properties of the Induced Norm)

Let $V$ be an inner product space over $F$ , $\mathbf{u}, \mathbf{v} \in V$ and $c \in F$ . Then:

(a) $\left\| \mathbf{v} \right\| \ge 0$ , with $\left\| \mathbf{v} \right\| = 0$ if and only if $\mathbf{v} = \mathbf{0}$ ;
(b) $\left\| c\mathbf{v} \right\| = \lvert c \rvert \left\| \mathbf{v} \right\|$ ;
(c) $\left\| \mathbf{u} + \mathbf{v} \right\|^2 = \left\| \mathbf{u} \right\|^2 + 2\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle + \left\| \mathbf{v} \right\|^2$ .

Proof.

(a) Immediate from Theorem 11.1.1 (a) and the fact that a non-negative real number has square root $0$ only when it is $0$ .
(b) By (IP1) and Theorem 11.1.1 (b), $\langle c\mathbf{v}, c\mathbf{v} \rangle = c\overline{c}\langle \mathbf{v}, \mathbf{v} \rangle = \lvert c \rvert^2\left\| \mathbf{v} \right\|^2$ using Theorem 0.6.2 (c). Taking non-negative square roots gives $\left\| c\mathbf{v} \right\| = \lvert c \rvert\left\| \mathbf{v} \right\|$ , since $\lvert c \rvert \ge 0$ .
(c) Expand one slot at a time, by (IP1) and then Theorem 11.1.1 (b): $\begin{aligned} \left\| \mathbf{u} + \mathbf{v} \right\|^2 &= \langle \mathbf{u} + \mathbf{v}, \mathbf{u} + \mathbf{v} \rangle \\ &= \langle \mathbf{u}, \mathbf{u} \rangle + \langle \mathbf{u}, \mathbf{v} \rangle + \langle \mathbf{v}, \mathbf{u} \rangle + \langle \mathbf{v}, \mathbf{v} \rangle . \end{aligned}$ By (IP2), $\langle \mathbf{v}, \mathbf{u} \rangle = \overline{\langle \mathbf{u}, \mathbf{v} \rangle}$ , and $z + \overline{z} = 2\operatorname{Re}z$ by Theorem 0.6.2 (c). This gives the stated identity.

Part (c) is the workhorse. Over $\mathbb{R}$ the middle term is just $2\langle \mathbf{u}, \mathbf{v} \rangle$ , and the identity is the familiar $(a + b)^2 = a^2 + 2ab + b^2$ for vectors. Replacing $\mathbf{v}$ by $-\mathbf{v}$ gives $\left\| \mathbf{u} - \mathbf{v} \right\|^2 = \left\| \mathbf{u} \right\|^2 - 2\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle + \left\| \mathbf{v} \right\|^2$ , and adding the two versions gives the parallelogram law $\left\| \mathbf{u} + \mathbf{v} \right\|^2 + \left\| \mathbf{u} - \mathbf{v} \right\|^2 = 2\left\| \mathbf{u} \right\|^2 + 2\left\| \mathbf{v} \right\|^2 . \tag{$\ast$}$ We return to $(\ast)$ in the last warning of this section.

Warning. A squared norm is real and non-negative, never complex. If a computation in a complex space produces $\left\| \mathbf{v} \right\|^2 = 2i$ , or a negative number, something has gone wrong — most often a missing conjugate. For $\mathbf{v} = (1, i) \in \mathbb{C}^2$ , the correct value is $\left\| \mathbf{v} \right\|^2 = \lvert 1 \rvert^2 + \lvert i \rvert^2 = 2$ , while the conjugate-free expression $\mathbf{v}^{\top}\mathbf{v}$ gives $0$ . Use the answer as a check on the work.

11.1.6. Orthogonality and Pythagoras

With a norm in hand, the special relation $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ deserves its name.

Definition 11.1.4 (Orthogonal Vectors)

Let $V$ be an inner product space. Vectors $\mathbf{u}, \mathbf{v} \in V$ are orthogonal, written $\mathbf{u} \perp \mathbf{v}$ , if $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ . A list $(\mathbf{v}_1, \dots, \mathbf{v}_k)$ is orthogonal if $\mathbf{v}_i \perp \mathbf{v}_j$ whenever $i \ne j$ .

The relation is symmetric: $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ if and only if $\langle \mathbf{v}, \mathbf{u} \rangle = \overline{\langle \mathbf{u}, \mathbf{v} \rangle} = 0$ , so “ $\mathbf{u} \perp \mathbf{v}$ ” needs no order. The zero vector is orthogonal to everything (Theorem 11.1.1 (c)), and by part (a) of the same result it is the only vector orthogonal to itself. In $\mathbb{R}^n$ this is perpendicularity in the usual sense, by the law of cosines computation above: $\cos\theta = 0$ exactly when $\mathbf{x} \cdot \mathbf{y} = 0$ . In $\mathbb{C}^2$ , the vectors $(1, i)$ and $(1, -i)$ are orthogonal, since $1\cdot\overline{1} + i\cdot\overline{(-i)} = 1 + i \cdot i = 0$ .

Theorem 11.1.3 (Pythagorean Theorem)

Let $(\mathbf{v}_1, \dots, \mathbf{v}_k)$ be an orthogonal list in an inner product space. Then $\left\| \mathbf{v}_1 + \dots + \mathbf{v}_k \right\|^2 = \left\| \mathbf{v}_1 \right\|^2 + \dots + \left\| \mathbf{v}_k \right\|^2 .$

Proof. Expand the left side by (IP1) and Theorem 11.1.1 (b): $\Big\langle \sum_{i} \mathbf{v}_i,\ \sum_{j} \mathbf{v}_j \Big\rangle = \sum_{i}\sum_{j} \langle \mathbf{v}_i, \mathbf{v}_j \rangle .$ Every term with $i \ne j$ is $0$ , because the list is orthogonal. What survives is $\sum_i \langle \mathbf{v}_i, \mathbf{v}_i \rangle = \sum_i \left\| \mathbf{v}_i \right\|^2$ , as claimed.

So orthogonality is exactly the condition under which squared lengths add. This is why the relation is worth naming, and why the rest of the chapter works so hard to produce orthogonal vectors.

Warning. Over $\mathbb{C}$ , the converse of Pythagoras fails. By Theorem 11.1.2 (c), $\left\| \mathbf{u} + \mathbf{v} \right\|^2 = \left\| \mathbf{u} \right\|^2 + \left\| \mathbf{v} \right\|^2$ says only that $\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle = 0$ , not that $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ . In $\mathbb{C}$ with $\langle z, w \rangle = z\overline{w}$ , take $\mathbf{u} = 1$ and $\mathbf{v} = i$ : then $\langle \mathbf{u}, \mathbf{v} \rangle = \overline{i} = -i \ne 0$ , yet $\left\| 1 + i \right\|^2 = 2 = \left\| 1 \right\|^2 + \left\| i \right\|^2$ . Over $\mathbb{R}$ the converse does hold, since there $\operatorname{Re}$ does nothing.

Theorem 11.1.4 (Orthogonal Non-zero Vectors Are Independent)

Let $(\mathbf{v}_1, \dots, \mathbf{v}_k)$ be an orthogonal list of non-zero vectors in an inner product space. Then the list is linearly independent.

Idea. Start a dependence relation in the standard way, and then do the one thing an inner product allows: pair the relation with a single $\mathbf{v}_j$ . Orthogonality kills every term except the $j$ -th, and what is left is $a_j\left\| \mathbf{v}_j \right\|^2 = 0$ with $\left\| \mathbf{v}_j \right\| \ne 0$ .

Proof. Let $a_1\mathbf{v}_1 + \dots + a_k\mathbf{v}_k = \mathbf{0}$ with $a_1, \dots, a_k \in F$ . Fix $j$ and take the inner product of both sides with $\mathbf{v}_j$ . By (IP1) and Theorem 11.1.1 (c), $0 = \Big\langle \sum_{i} a_i\mathbf{v}_i,\ \mathbf{v}_j \Big\rangle = \sum_{i} a_i\langle \mathbf{v}_i, \mathbf{v}_j \rangle = a_j\left\| \mathbf{v}_j \right\|^2 ,$ where the last equality holds because $\langle \mathbf{v}_i, \mathbf{v}_j \rangle = 0$ for $i \ne j$ . Since $\mathbf{v}_j \ne \mathbf{0}$ , Theorem 11.1.1 (a) gives $\left\| \mathbf{v}_j \right\|^2 > 0$ , so $a_j = 0$ . As $j$ was arbitrary, all the coefficients vanish, and the list is linearly independent by Definition 1.4.1.

The hypothesis “non-zero” cannot be dropped: the list $(\mathbf{e}_1, \mathbf{0})$ is orthogonal and dependent. With it, the result is a free supply of independent lists, and this is what makes the next section possible.

11.1.7. The Cauchy–Schwarz inequality

Angles came from the formula $\cos\theta = (\mathbf{x} \cdot \mathbf{y})/(\left\| \mathbf{x} \right\|\left\| \mathbf{y} \right\|)$ . To define an angle by that formula in an abstract space, we must know that the right-hand side lies in $[-1, 1]$ . That is exactly the following inequality, the most used inequality in the subject.

Theorem 11.1.5 (Cauchy–Schwarz Inequality)

Let $V$ be an inner product space and $\mathbf{u}, \mathbf{v} \in V$ . Then $\lvert \langle \mathbf{u}, \mathbf{v} \rangle \rvert \le \left\| \mathbf{u} \right\|\,\left\| \mathbf{v} \right\| ,$ with equality if and only if the list $(\mathbf{u}, \mathbf{v})$ is linearly dependent.

Idea. Picture $\mathbf{u}$ and $\mathbf{v}$ in $\mathbb{R}^2$ . Drop a perpendicular from the tip of $\mathbf{u}$ onto the line through $\mathbf{v}$ . The foot of the perpendicular is a multiple $c\mathbf{v}$ , and the two legs $c\mathbf{v}$ and $\mathbf{u} - c\mathbf{v}$ of the resulting right triangle have squared lengths adding to $\left\| \mathbf{u} \right\|^2$ . Since one leg has squared length $\lvert\langle \mathbf{u}, \mathbf{v} \rangle\rvert^2 / \left\| \mathbf{v} \right\|^2$ , the inequality is just “a leg is no longer than the hypotenuse”, and equality means the other leg is $\mathbf{0}$ . The picture proves nothing in an abstract space, so we check algebraically that $\mathbf{u} - c\mathbf{v} \perp \mathbf{v}$ for the right $c$ , and then let Pythagoras do the work.

Proof. If $\mathbf{v} = \mathbf{0}$ , both sides are $0$ by Theorem 11.1.1 (c), and $(\mathbf{u}, \mathbf{0})$ is dependent, so the statement holds. Suppose $\mathbf{v} \ne \mathbf{0}$ , so $\left\| \mathbf{v} \right\| > 0$ , and put $c = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{\left\| \mathbf{v} \right\|^2}, \qquad \mathbf{z} = \mathbf{u} - c\mathbf{v} .$ Then $\mathbf{z} \perp \mathbf{v}$ , since by (IP1) $\langle \mathbf{z}, \mathbf{v} \rangle = \langle \mathbf{u}, \mathbf{v} \rangle - c\langle \mathbf{v}, \mathbf{v} \rangle = \langle \mathbf{u}, \mathbf{v} \rangle - \langle \mathbf{u}, \mathbf{v} \rangle = 0 ,$ and hence also $\mathbf{z} \perp c\mathbf{v}$ . Now $\mathbf{u} = \mathbf{z} + c\mathbf{v}$ is a sum of two orthogonal vectors, so Theorem 11.1.3 and Theorem 11.1.2 (b) give $\left\| \mathbf{u} \right\|^2 = \left\| \mathbf{z} \right\|^2 + \lvert c \rvert^2\left\| \mathbf{v} \right\|^2 = \left\| \mathbf{z} \right\|^2 + \frac{\lvert \langle \mathbf{u}, \mathbf{v} \rangle \rvert^2}{\left\| \mathbf{v} \right\|^2} . \tag{$\dagger$}$ Since $\left\| \mathbf{z} \right\|^2 \ge 0$ , dropping it gives $\left\| \mathbf{u} \right\|^2\left\| \mathbf{v} \right\|^2 \ge \lvert \langle \mathbf{u}, \mathbf{v} \rangle \rvert^2$ , and taking non-negative square roots proves the inequality.

Equality holds in $(\dagger)$ exactly when $\left\| \mathbf{z} \right\| = 0$ , that is (Theorem 11.1.2 (a)) exactly when $\mathbf{u} = c\mathbf{v}$ . If $\mathbf{u} = c\mathbf{v}$ , the list $(\mathbf{u}, \mathbf{v})$ is dependent. Conversely, suppose $(\mathbf{u}, \mathbf{v})$ is dependent, say $a\mathbf{u} + b\mathbf{v} = \mathbf{0}$ with $a, b$ not both $0$ . Here $a \ne 0$ , since $a = 0$ would give $b\mathbf{v} = \mathbf{0}$ with $b \ne 0$ and hence $\mathbf{v} = \mathbf{0}$ , which we have excluded. So $\mathbf{u} = d\mathbf{v}$ with $d = -b/a$ , and then $c = d\langle \mathbf{v}, \mathbf{v} \rangle/\left\| \mathbf{v} \right\|^2 = d$ , so $\mathbf{z} = \mathbf{0}$ . This proves the equality case.

The proof is worth rereading for its shape: the vector $\mathbf{z}$ was found from a picture and then checked from the axioms. That is the standard way to work in an abstract inner product space.

Corollary 11.1.6 (Triangle Inequality)

Let $V$ be an inner product space and $\mathbf{u}, \mathbf{v} \in V$ . Then $\left\| \mathbf{u} + \mathbf{v} \right\| \le \left\| \mathbf{u} \right\| + \left\| \mathbf{v} \right\| .$

Proof. For any complex number $z$ we have $\operatorname{Re}z \le \lvert z \rvert$ , as in the proof of Theorem 0.6.4. Hence, by Theorem 11.1.2 (c) and then Theorem 11.1.5, $\begin{aligned} \left\| \mathbf{u} + \mathbf{v} \right\|^2 &= \left\| \mathbf{u} \right\|^2 + 2\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle + \left\| \mathbf{v} \right\|^2 \\ &\le \left\| \mathbf{u} \right\|^2 + 2\lvert\langle \mathbf{u}, \mathbf{v} \rangle\rvert + \left\| \mathbf{v} \right\|^2 \\ &\le \left\| \mathbf{u} \right\|^2 + 2\left\| \mathbf{u} \right\|\left\| \mathbf{v} \right\| + \left\| \mathbf{v} \right\|^2 \\ &= \big(\left\| \mathbf{u} \right\| + \left\| \mathbf{v} \right\|\big)^2 . \end{aligned}$ Both $\left\| \mathbf{u} + \mathbf{v} \right\|$ and $\left\| \mathbf{u} \right\| + \left\| \mathbf{v} \right\|$ are $\ge 0$ , so taking square roots preserves the inequality.

Together, Theorem 11.1.2 (a), (b) and Corollary 11.1.6 say that $\left\| \cdot \right\|$ is a norm in the general sense studied in Chapter 16. Chapter 16 also shows that not every norm arises from an inner product; the last warning of this section explains how to tell.

Definition 11.1.5 (Angle Between Vectors)

Let $V$ be a real inner product space and let $\mathbf{u}, \mathbf{v} \in V$ be non-zero. The angle between $\mathbf{u}$ and $\mathbf{v}$ is the unique $\theta \in [0, \pi]$ with $\cos\theta = \frac{\langle \mathbf{u}, \mathbf{v} \rangle}{\left\| \mathbf{u} \right\|\,\left\| \mathbf{v} \right\|} .$

The definition needs Theorem 11.1.5 twice over. Cauchy–Schwarz says the quotient has absolute value at most $1$ , so it lies in $[-1, 1]$ ; and $\cos$ is a strictly decreasing bijection from $[0, \pi]$ onto $[-1, 1]$ , so exactly one $\theta$ qualifies. Without the inequality the formula could ask for the angle whose cosine is $7$ . Note also that $\theta = \pi/2$ if and only if $\langle \mathbf{u}, \mathbf{v} \rangle = 0$ , so the definition is consistent with Definition 11.1.4.

Over $\mathbb{C}$ there is no angle, because $\langle \mathbf{u}, \mathbf{v} \rangle$ is not real. The quantity $\lvert\langle \mathbf{u}, \mathbf{v} \rangle\rvert / (\left\| \mathbf{u} \right\|\left\| \mathbf{v} \right\|) \in [0, 1]$ still makes sense, and it still measures how far the two vectors are from being orthogonal.

Example (Three Angles)

Compute the angle between $\mathbf{u} = (1, 1, 0)$ and $\mathbf{v} = (1, 0, 1)$ in $\mathbb{R}^3$ with the dot product; between $p = 1$ and $q = x$ in $\mathbb{R}[x]_{\le 1}$ with $\langle p, q \rangle = \int_0^1 pq$ ; and between $\mathbf{A} = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}$ and $\mathbf{B} = \begin{pmatrix} 1 & -1 \\ 1 & -1 \end{pmatrix}$ in $M_2(\mathbb{R})$ with the Frobenius inner product.

Solution. In $\mathbb{R}^3$ : $\langle \mathbf{u}, \mathbf{v} \rangle = 1$ , $\left\| \mathbf{u} \right\| = \left\| \mathbf{v} \right\| = \sqrt2$ , so $\cos\theta = 1/2$ and $\theta = \pi/3$ .

In $\mathbb{R}[x]_{\le 1}$ : $\langle 1, x \rangle = \int_0^1 t\,\mathrm{d} t = \tfrac12$ , $\left\| 1 \right\|^2 = \int_0^1 1 = 1$ and $\left\| x \right\|^2 = \int_0^1 t^2\,\mathrm{d} t = \tfrac13$ . Hence $\cos\theta = \frac{1/2}{1 \cdot 1/\sqrt3} = \frac{\sqrt3}{2},$ so $\theta = \pi/6$ . The constant $1$ and the function $x$ are far from orthogonal on $[0, 1]$ , which is believable: both are positive there.

In $M_2(\mathbb{R})$ : $\langle \mathbf{A}, \mathbf{B} \rangle = 1 - 1 + 1 - 1 = 0$ , so $\theta = \pi/2$ . The two matrices are orthogonal, with $\left\| \mathbf{A} \right\| = \left\| \mathbf{B} \right\| = 2$ . As a check on Pythagoras, $\mathbf{A} + \mathbf{B} = \begin{pmatrix} 2 & 0 \\ 2 & 0 \end{pmatrix}$ has $\left\| \mathbf{A} + \mathbf{B} \right\|^2 = 8 = 4 + 4$ .

Quick check. In $\mathbb{C}^2$ with the standard inner product, compute $\langle (1, i), (1, 1) \rangle$ and $\langle (1, 1), (1, i) \rangle$ . Check the answers against (IP2), and against Cauchy–Schwarz.

Solution. $\langle (1,i), (1,1) \rangle = 1\cdot\conj1 + i\cdot\conj1 = 1 + i$ , and $\langle (1,1), (1,i) \rangle = 1\cdot\conj1 + 1\cdot\overline{i} = 1 - i$ . They are conjugates, as (IP2) requires. For Cauchy–Schwarz, $\lvert 1 + i \rvert = \sqrt2$ , while $\left\| (1,i) \right\| = \sqrt2$ and $\left\| (1,1) \right\| = \sqrt2$ , so the inequality reads $\sqrt2 \le 2$ . It is strict, which is right: $(1, i)$ and $(1, 1)$ are independent.

Warning. Not every norm comes from an inner product. A function $\left\| \cdot \right\|$ can satisfy positivity, homogeneity and the inequality of Corollary 11.1.6 without being $\sqrt{\langle \mathbf{v}, \mathbf{v} \rangle}$ for any inner product. The test is the parallelogram law $(\ast)$ : every induced norm satisfies it, as we derived from Theorem 11.1.2 (c), so a norm that violates it is induced by nothing. Take $\left\| \mathbf{x} \right\|_\infty = \max(\lvert x_1 \rvert, \lvert x_2 \rvert)$ on $\mathbb{R}^2$ with $\mathbf{u} = \mathbf{e}_1$ , $\mathbf{v} = \mathbf{e}_2$ . Then $\left\| \mathbf{u} + \mathbf{v} \right\|_\infty = \left\| \mathbf{u} - \mathbf{v} \right\|_\infty = 1$ , so the left side of $(\ast)$ is $2$ , while the right side is $4$ . Remarkably, the law is not only necessary but sufficient; Exercise (C1: The parallelogram law characterizes inner-product norms) proves the substance of that in $\mathbb{R}^2$ , and Chapter 16 takes up general norms.

11.1.8. Exercises

A. Check your understanding

Exercise (A1)

(a) State the three axioms of an inner product on a vector space over $F = \mathbb{R}$ or $\mathbb{C}$ , and say in which slot the map is linear.
(b) Why must the field be $\mathbb{R}$ or $\mathbb{C}$ ? Name the two features used.
(c) Explain why $\langle \mathbf{v}, \mathbf{v} \rangle$ is always a real number.
(d) True or false: an inner product on a complex space is linear in the second slot. Justify your answer.
(e) State the Cauchy–Schwarz inequality, its equality case, and one place where the equality case is used.
(f) True or false: if $\left\| \mathbf{u} + \mathbf{v} \right\|^2 = \left\| \mathbf{u} \right\|^2 + \left\| \mathbf{v} \right\|^2$ in a complex inner product space, then $\mathbf{u} \perp \mathbf{v}$ . Justify your answer.

Solution.

(a) (IP1) $\langle a\mathbf{u} + b\mathbf{v}, \mathbf{w} \rangle = a\langle \mathbf{u}, \mathbf{w} \rangle + b\langle \mathbf{v}, \mathbf{w} \rangle$ ; (IP2) $\langle \mathbf{v}, \mathbf{u} \rangle = \overline{\langle \mathbf{u}, \mathbf{v} \rangle}$ ; (IP3) $\langle \mathbf{v}, \mathbf{v} \rangle > 0$ for $\mathbf{v} \ne \mathbf{0}$ (Definition 11.1.1). It is linear in the first slot.
(b) Positivity, so that $\langle \mathbf{v}, \mathbf{v} \rangle > 0$ has meaning and square roots exist, and conjugation, so that $\langle \mathbf{v}, \mathbf{v} \rangle$ is real and non-negative over $\mathbb{C}$ .
(c) By (IP2) with $\mathbf{u} = \mathbf{v}$ , the number equals its own conjugate, so its imaginary part is $0$ (Theorem 11.1.1 (a)).
(d) False. It is conjugate-linear in the second slot: $\langle \mathbf{u}, a\mathbf{v} \rangle = \overline{a}\langle \mathbf{u}, \mathbf{v} \rangle$ (Theorem 11.1.1 (b)). For example, in $\mathbb{C}$ with $\langle z, w \rangle = z\overline{w}$ , $\langle 1, i \rangle = -i$ while $i\langle 1, 1 \rangle = i$ .
(e) $\lvert\langle \mathbf{u}, \mathbf{v} \rangle\rvert \le \left\| \mathbf{u} \right\|\left\| \mathbf{v} \right\|$ , with equality if and only if $(\mathbf{u}, \mathbf{v})$ is dependent (Theorem 11.1.5). The inequality is what makes Definition 11.1.5 well posed: it puts the quotient $\langle \mathbf{u}, \mathbf{v} \rangle/(\left\| \mathbf{u} \right\|\left\| \mathbf{v} \right\|)$ inside $[-1, 1]$ , where $\arccos$ is defined. The equality case then says which vectors sit at the ends of that interval: $\cos\theta = \pm 1$ exactly for dependent pairs, that is, $\theta = 0$ or $\theta = \pi$ .
(f) False. It says only $\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle = 0$ . In $\mathbb{C}$ with $\langle z, w \rangle = z\overline{w}$ , take $\mathbf{u} = 1$ , $\mathbf{v} = i$ : then $\langle \mathbf{u}, \mathbf{v} \rangle = -i \ne 0$ but $\left\| 1 + i \right\|^2 = 2 = 1 + 1$ .

B. Practice

Exercise (B1: Determine which are inner products)

Determine which of the following are inner products on the stated space. For those that are, verify the axioms; for those that are not, name the clause that fails and give an explicit witness.

(a) On $\mathbb{R}^2$ : $\langle \mathbf{x}, \mathbf{y} \rangle = x_1y_1 + 3x_2y_2$ .
(b) On $\mathbb{R}^2$ : $\langle \mathbf{x}, \mathbf{y} \rangle = x_1y_1 + x_1y_2 + x_2y_1 + 2x_2y_2$ .
(c) On $\mathbb{C}^2$ : $\langle \mathbf{x}, \mathbf{y} \rangle = x_1\overline{y_1} + x_2y_2$ .
(d) On $M_2(\mathbb{R})$ : $\langle \mathbf{A}, \mathbf{B} \rangle = \operatorname{tr}(\mathbf{A}\mathbf{B})$ .
(e) On $\mathbb{R}[x]_{\le 2}$ : $\langle p, q \rangle = p(0)q(0) + p(1)q(1) + p(2)q(2)$ .

Solution.

(a) An inner product: it is the weighted product with weights $1, 3 > 0$ . Bilinearity and symmetry are clear, and $\langle \mathbf{x}, \mathbf{x} \rangle = x_1^2 + 3x_2^2 = 0$ forces $x_1 = x_2 = 0$ .
(b) An inner product. It is bilinear term by term, and symmetric because the two cross terms are interchanged by swapping $\mathbf{x}$ and $\mathbf{y}$ . For positivity, complete the square: $\langle \mathbf{x}, \mathbf{x} \rangle = x_1^2 + 2x_1x_2 + 2x_2^2 = (x_1 + x_2)^2 + x_2^2 ,$ which is $0$ only if $x_2 = 0$ and $x_1 + x_2 = 0$ , that is $\mathbf{x} = \mathbf{0}$ .
(c) Not an inner product. (IP2) fails: with $\mathbf{x} = (0, 1)$ and $\mathbf{y} = (0, i)$ we get $\langle \mathbf{x}, \mathbf{y} \rangle = i$ and $\langle \mathbf{y}, \mathbf{x} \rangle = i$ , while (IP2) would require $\langle \mathbf{y}, \mathbf{x} \rangle = \overline{i} = -i$ . Positivity fails too, at $\mathbf{x} = (0, i)$ : $\langle \mathbf{x}, \mathbf{x} \rangle = i^2 = -1$ .
(d) Not an inner product. It is bilinear, and symmetric by Theorem 0.9.6 (3). But (IP3) fails for $\mathbf{N} = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix} \ne 0$ , since $\mathbf{N}^2 = 0$ and so $\langle \mathbf{N}, \mathbf{N} \rangle = \operatorname{tr}(\mathbf{N}^2) = 0$ . Replacing $\operatorname{tr}(\mathbf{A}\mathbf{B})$ by $\operatorname{tr}(\mathbf{B}^{\top}\mathbf{A})$ repairs it: that is the Frobenius inner product.
(e) An inner product: the node example with the three distinct nodes $0, 1, 2$ on $\mathbb{R}[x]_{\le 2}$ . Bilinearity and symmetry come from linearity of evaluation. If $\langle p, p \rangle = p(0)^2 + p(1)^2 + p(2)^2 = 0$ , then $p$ has three distinct roots and degree at most $2$ , so $p = 0$ by Corollary 6.1.4.

Exercise (B2: Norms and angles)

(a) In $\mathbb{R}^3$ with the dot product, find $\left\| \mathbf{u} \right\|$ , $\left\| \mathbf{v} \right\|$ and the cosine of the angle between $\mathbf{u} = (1, 2, 2)$ and $\mathbf{v} = (2, -1, 2)$ .
(b) In $\mathbb{R}[x]_{\le 2}$ with $\langle p, q \rangle = \int_{-1}^{1} pq$ , compute $\left\| 1 \right\|$ , $\left\| x \right\|$ and $\langle 1, x \rangle$ . What is the angle between $1$ and $x$ ?
(c) In $M_2(\mathbb{C})$ with the Frobenius inner product, compute $\left\| \mathbf{A} \right\|$ for $\mathbf{A} = \begin{pmatrix} 1 & i \\ 0 & 1 + i \end{pmatrix}$ .

Solution.

(a) $\left\| \mathbf{u} \right\| = \sqrt{1 + 4 + 4} = 3$ and $\left\| \mathbf{v} \right\| = \sqrt{4 + 1 + 4} = 3$ . Also $\langle \mathbf{u}, \mathbf{v} \rangle = 2 - 2 + 4 = 4$ , so $\cos\theta = 4/9$ . Cauchy–Schwarz reads $4 \le 9$ , and it is strict, as it must be for an independent pair.
(b) $\left\| 1 \right\|^2 = \int_{-1}^1 1\,\mathrm{d} t = 2$ , so $\left\| 1 \right\| = \sqrt2$ ; $\left\| x \right\|^2 = \int_{-1}^1 t^2\,\mathrm{d} t = \tfrac23$ , so $\left\| x \right\| = \sqrt{2/3}$ ; and $\langle 1, x \rangle = \int_{-1}^1 t\,\mathrm{d} t = 0$ . The angle is $\pi/2$ : on a symmetric interval, the constants are orthogonal to the odd functions. Compare this with Example (Three Angles), where the interval $[0, 1]$ gave the angle $\pi/6$ instead. The geometry depends on the inner product, not only on the vectors.
(c) $\left\| \mathbf{A} \right\|^2 = \lvert 1 \rvert^2 + \lvert i \rvert^2 + 0 + \lvert 1 + i \rvert^2 = 1 + 1 + 2 = 4$ , so $\left\| \mathbf{A} \right\| = 2$ .

Exercise (B3: Cauchy–Schwarz, concretely)

(a) Verify Theorem 11.1.5 for $\mathbf{a} = (1, 2, 3)$ and $\mathbf{b} = (3, 2, 1)$ in $\mathbb{R}^3$ with the dot product, and say whether the inequality is strict.
(b) Deduce from Theorem 11.1.5 the classical inequality $\Big(\sum_{i=1}^{n} a_ib_i\Big)^2 \le \Big(\sum_{i=1}^{n} a_i^2\Big)\Big(\sum_{i=1}^{n} b_i^2\Big)$ for all real $a_i, b_i$ .
(c) Hence show that $(a_1 + \dots + a_n)^2 \le n(a_1^2 + \dots + a_n^2)$ for all real $a_i$ .

Solution.

(a) $\langle \mathbf{a}, \mathbf{b} \rangle = 3 + 4 + 3 = 10$ , so $\lvert\langle \mathbf{a}, \mathbf{b} \rangle\rvert^2 = 100$ , while $\left\| \mathbf{a} \right\|^2\left\| \mathbf{b} \right\|^2 = 14 \cdot 14 = 196$ . So $10 \le 14$ , strictly. It must be strict, since $\mathbf{a}$ and $\mathbf{b}$ are not multiples of each other.
(b) Apply Theorem 11.1.5 on $\mathbb{R}^n$ with the dot product to $\mathbf{a} = (a_1, \dots, a_n)$ and $\mathbf{b} = (b_1, \dots, b_n)$ . Then $\langle \mathbf{a}, \mathbf{b} \rangle = \sum a_ib_i$ , $\left\| \mathbf{a} \right\|^2 = \sum a_i^2$ and $\left\| \mathbf{b} \right\|^2 = \sum b_i^2$ , and squaring the inequality $\lvert\langle \mathbf{a}, \mathbf{b} \rangle\rvert \le \left\| \mathbf{a} \right\|\left\| \mathbf{b} \right\|$ gives the claim.
(c) Take $b_i = 1$ for all $i$ in (b). Then $\sum a_ib_i = \sum a_i$ and $\sum b_i^2 = n$ , so $(\sum a_i)^2 \le n\sum a_i^2$ . Equality holds exactly when $\mathbf{a}$ is a multiple of $(1, \dots, 1)$ , that is, when all the $a_i$ are equal.

Exercise (B4: A lower bound for a difference of norms)

Let $V$ be an inner product space and $\mathbf{u}, \mathbf{v} \in V$ . Prove that $\big\lvert \left\| \mathbf{u} \right\| - \left\| \mathbf{v} \right\| \big\rvert \le \left\| \mathbf{u} - \mathbf{v} \right\| .$

Solution. Write $\mathbf{u} = (\mathbf{u} - \mathbf{v}) + \mathbf{v}$ . By Corollary 11.1.6, $\left\| \mathbf{u} \right\| \le \left\| \mathbf{u} - \mathbf{v} \right\| + \left\| \mathbf{v} \right\|,$ so $\left\| \mathbf{u} \right\| - \left\| \mathbf{v} \right\| \le \left\| \mathbf{u} - \mathbf{v} \right\|$ . Swapping the roles of $\mathbf{u}$ and $\mathbf{v}$ gives $\left\| \mathbf{v} \right\| - \left\| \mathbf{u} \right\| \le \left\| \mathbf{v} - \mathbf{u} \right\|$ , and $\left\| \mathbf{v} - \mathbf{u} \right\| = \lvert -1 \rvert\left\| \mathbf{u} - \mathbf{v} \right\| = \left\| \mathbf{u} - \mathbf{v} \right\|$ by Theorem 11.1.2 (b). The quantity $\lvert \left\| \mathbf{u} \right\| - \left\| \mathbf{v} \right\| \rvert$ is one of the two numbers just bounded, so it is at most $\left\| \mathbf{u} - \mathbf{v} \right\|$ . This proves the inequality.

C. Going deeper

Exercise (C1: The parallelogram law characterizes inner-product norms)

Let $\left\| \cdot \right\|$ be a norm on $\mathbb{R}^2$ , that is, a function $\mathbb{R}^2 \to \mathbb{R}$ satisfying the three properties of Theorem 11.1.2 (a), (b) and Corollary 11.1.6. Suppose it satisfies the parallelogram law $\left\| \mathbf{u} + \mathbf{v} \right\|^2 + \left\| \mathbf{u} - \mathbf{v} \right\|^2 = 2\left\| \mathbf{u} \right\|^2 + 2\left\| \mathbf{v} \right\|^2$ for all $\mathbf{u}, \mathbf{v}$ , and define $\beta(\mathbf{u}, \mathbf{v}) = \tfrac14\big(\left\| \mathbf{u} + \mathbf{v} \right\|^2 - \left\| \mathbf{u} - \mathbf{v} \right\|^2\big)$ .

(a) Show that $\beta(\mathbf{u}, \mathbf{u}) = \left\| \mathbf{u} \right\|^2$ , that $\beta(\mathbf{u}, \mathbf{v}) = \beta(\mathbf{v}, \mathbf{u})$ , and that $\beta(\mathbf{u}, \mathbf{0}) = 0$ .
(b) Prove that $\beta(\mathbf{u}, \mathbf{v}) + \beta(\mathbf{u}, \mathbf{w}) = 2\beta\big(\mathbf{u}, \tfrac12(\mathbf{v} + \mathbf{w})\big)$ for all $\mathbf{u}, \mathbf{v}, \mathbf{w}$ .
(c) Deduce that $\beta(\mathbf{u}, \mathbf{v}) + \beta(\mathbf{u}, \mathbf{w}) = \beta(\mathbf{u}, \mathbf{v} + \mathbf{w})$ .
(d) Granting that $\beta(\mathbf{u}, t\mathbf{v}) = t\beta(\mathbf{u}, \mathbf{v})$ for all $t \in \mathbb{R}$ , conclude that $\beta$ is an inner product on $\mathbb{R}^2$ whose induced norm is $\left\| \cdot \right\|$ .

Hint: for (b), apply the parallelogram law to the pair $\mathbf{u} + \tfrac12(\mathbf{v} + \mathbf{w})$ , $\tfrac12(\mathbf{v} - \mathbf{w})$ , and then to the pair $\mathbf{u} - \tfrac12(\mathbf{v} + \mathbf{w})$ , $\tfrac12(\mathbf{w} - \mathbf{v})$ .

Solution.

(a) $\beta(\mathbf{u}, \mathbf{u}) = \tfrac14(\left\| 2\mathbf{u} \right\|^2 - \left\| \mathbf{0} \right\|^2) = \tfrac14 \cdot 4\left\| \mathbf{u} \right\|^2 = \left\| \mathbf{u} \right\|^2$ , using homogeneity. Symmetry holds because $\left\| \mathbf{v} - \mathbf{u} \right\| = \left\| \mathbf{u} - \mathbf{v} \right\|$ , again by homogeneity with $c = -1$ . Finally $\beta(\mathbf{u}, \mathbf{0}) = \tfrac14(\left\| \mathbf{u} \right\|^2 - \left\| \mathbf{u} \right\|^2) = 0$ .
(b) Put $\mathbf{m} = \tfrac12(\mathbf{v} + \mathbf{w})$ and $\mathbf{d} = \tfrac12(\mathbf{v} - \mathbf{w})$ , so that $\mathbf{v} = \mathbf{m} + \mathbf{d}$ and $\mathbf{w} = \mathbf{m} - \mathbf{d}$ . Apply the parallelogram law to $\mathbf{u} + \mathbf{m}$ and $\mathbf{d}$ : since $(\mathbf{u} + \mathbf{m}) + \mathbf{d} = \mathbf{u} + \mathbf{v}$ and $(\mathbf{u} + \mathbf{m}) - \mathbf{d} = \mathbf{u} + \mathbf{w}$ , $\left\| \mathbf{u} + \mathbf{v} \right\|^2 + \left\| \mathbf{u} + \mathbf{w} \right\|^2 = 2\left\| \mathbf{u} + \mathbf{m} \right\|^2 + 2\left\| \mathbf{d} \right\|^2 .$ Apply it to $\mathbf{u} - \mathbf{m}$ and $\mathbf{d}$ : since $(\mathbf{u} - \mathbf{m}) - \mathbf{d} = \mathbf{u} - \mathbf{v}$ and $(\mathbf{u} - \mathbf{m}) + \mathbf{d} = \mathbf{u} - \mathbf{w}$ , $\left\| \mathbf{u} - \mathbf{v} \right\|^2 + \left\| \mathbf{u} - \mathbf{w} \right\|^2 = 2\left\| \mathbf{u} - \mathbf{m} \right\|^2 + 2\left\| \mathbf{d} \right\|^2 .$ Subtracting the second from the first, the terms $2\left\| \mathbf{d} \right\|^2$ cancel, and dividing by $4$ gives $\beta(\mathbf{u}, \mathbf{v}) + \beta(\mathbf{u}, \mathbf{w}) = 2\beta(\mathbf{u}, \mathbf{m})$ , which is the claim.
(c) Take $\mathbf{w} = \mathbf{0}$ in (b). By (a), $\beta(\mathbf{u}, \mathbf{0}) = 0$ , so $\beta(\mathbf{u}, \mathbf{v}) = 2\beta(\mathbf{u}, \tfrac12\mathbf{v})$ for all $\mathbf{v}$ . Applying this with $\mathbf{v} + \mathbf{w}$ in place of $\mathbf{v}$ , we get $2\beta(\mathbf{u}, \tfrac12(\mathbf{v} + \mathbf{w})) = \beta(\mathbf{u}, \mathbf{v} + \mathbf{w})$ , and the left side is $\beta(\mathbf{u}, \mathbf{v}) + \beta(\mathbf{u}, \mathbf{w})$ by (b).
(d) By (c) and the granted homogeneity, $\beta$ is linear in its second slot, and by the symmetry in (a) it is then linear in the first as well, so (IP1) and (IP2) hold (conjugation is trivial over $\mathbb{R}$ ). For (IP3), $\beta(\mathbf{u}, \mathbf{u}) = \left\| \mathbf{u} \right\|^2 > 0$ for $\mathbf{u} \ne \mathbf{0}$ , since a norm vanishes only at $\mathbf{0}$ . So $\beta$ is an inner product, and its induced norm is $\sqrt{\beta(\mathbf{u},\mathbf{u})} = \left\| \mathbf{u} \right\|$ .

Remark. The granted step in (d) is where analysis enters: additivity from (c) gives $\beta(\mathbf{u}, t\mathbf{v}) = t\beta(\mathbf{u}, \mathbf{v})$ for every rational $t$ by induction, and the extension to real $t$ uses that $t \mapsto \beta(\mathbf{u}, t\mathbf{v})$ is continuous, which follows from continuity of the norm. Nothing else in the argument needs anything beyond the three norm properties.

Exercise (C2: Recovering the inner product from the norm)

Let $V$ be an inner product space over $F$ .

(a) For $F = \mathbb{R}$ , prove the polarization identity $\langle \mathbf{u}, \mathbf{v} \rangle = \tfrac14\big(\left\| \mathbf{u} + \mathbf{v} \right\|^2 - \left\| \mathbf{u} - \mathbf{v} \right\|^2\big)$ .
(b) For $F = \mathbb{C}$ , prove that $\langle \mathbf{u}, \mathbf{v} \rangle = \frac14\sum_{k=0}^{3} i^{k}\,\left\| \mathbf{u} + i^{k}\mathbf{v} \right\|^2 .$
(c) Deduce that two inner products on $V$ with the same induced norm are equal.

Solution.

(a) By Theorem 11.1.2 (c) and its version with $-\mathbf{v}$ , and since $\operatorname{Re}$ is the identity on $\mathbb{R}$ , $\left\| \mathbf{u} + \mathbf{v} \right\|^2 - \left\| \mathbf{u} - \mathbf{v} \right\|^2 = 4\langle \mathbf{u}, \mathbf{v} \rangle.$ Dividing by $4$ gives the identity.
(b) Write $z = \langle \mathbf{u}, \mathbf{v} \rangle$ . By Theorem 11.1.2 (c) and Theorem 11.1.1 (b), $\left\| \mathbf{u} + i^k\mathbf{v} \right\|^2 = \left\| \mathbf{u} \right\|^2 + \left\| \mathbf{v} \right\|^2 + 2\operatorname{Re}\big(\overline{i^{k}}z\big),$ where we used $\left\| i^k\mathbf{v} \right\| = \lvert i^k \rvert \left\| \mathbf{v} \right\| = \left\| \mathbf{v} \right\|$ . Now $\sum_{k=0}^3 i^k = 1 + i - 1 - i = 0$ , so the terms $\left\| \mathbf{u} \right\|^2 + \left\| \mathbf{v} \right\|^2$ contribute nothing. Writing $z = a + bi$ with $a, b \in \mathbb{R}$ , the four remaining terms $2 i^k \operatorname{Re}(\overline{i^k}z)$ are, for $k = 0, 1, 2, 3$ , $2a, \quad 2ib, \quad 2a, \quad 2ib,$ since $\operatorname{Re}(z) = a$ , $\operatorname{Re}(-iz) = b$ , $\operatorname{Re}(-z) = -a$ and $\operatorname{Re}(iz) = -b$ . Their sum is $4a + 4bi = 4z$ , and dividing by $4$ gives the identity.
(c) Both (a) and (b) express $\langle \mathbf{u}, \mathbf{v} \rangle$ using only the norm. So if two inner products induce the same norm, the right-hand sides agree for all $\mathbf{u}, \mathbf{v}$ , hence so do the left-hand sides.

Exercise (C3: The equality cases)

Let $V$ be an inner product space.

(a) Prove that $\left\| \mathbf{u} + \mathbf{v} \right\| = \left\| \mathbf{u} \right\| + \left\| \mathbf{v} \right\|$ if and only if one of $\mathbf{u}, \mathbf{v}$ is a non-negative real multiple of the other.
(b) Deduce the equality condition in the classical inequality of Exercise (B3: Cauchy–Schwarz, concretely) (b): for real $a_i, b_i$ , equality holds if and only if one of the vectors $(a_1, \dots, a_n)$ , $(b_1, \dots, b_n)$ is a multiple of the other.
(c) Give real numbers $a_i, b_i$ , not all zero, with $\sum a_ib_i < 0$ and equality in the classical inequality, and explain why this does not contradict (a).

Solution.

(a) $(\Leftarrow)$ If $\mathbf{v} = \mathbf{0}$ both sides are $\left\| \mathbf{u} \right\|$ . If $\mathbf{u} = t\mathbf{v}$ with $t \ge 0$ , then $\left\| \mathbf{u} + \mathbf{v} \right\| = (t + 1)\left\| \mathbf{v} \right\| = t\left\| \mathbf{v} \right\| + \left\| \mathbf{v} \right\| = \left\| \mathbf{u} \right\| + \left\| \mathbf{v} \right\|$ by Theorem 11.1.2 (b).
$(\Rightarrow)$ Suppose $\left\| \mathbf{u} + \mathbf{v} \right\| = \left\| \mathbf{u} \right\| + \left\| \mathbf{v} \right\|$ . Squaring and using Theorem 11.1.2 (c), $2\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle = 2\left\| \mathbf{u} \right\|\left\| \mathbf{v} \right\| .$ The chain in the proof of Corollary 11.1.6 is therefore an equality throughout, so $\operatorname{Re}\langle \mathbf{u}, \mathbf{v} \rangle = \lvert\langle \mathbf{u}, \mathbf{v} \rangle\rvert = \left\| \mathbf{u} \right\|\left\| \mathbf{v} \right\|$ . The second equality and Theorem 11.1.5 make $(\mathbf{u}, \mathbf{v})$ dependent. Assume $\mathbf{v} \ne \mathbf{0}$ (otherwise $\mathbf{v} = 0\mathbf{u}$ and we are done) and write $\mathbf{u} = c\mathbf{v}$ . Then $\langle \mathbf{u}, \mathbf{v} \rangle = c\left\| \mathbf{v} \right\|^2$ , and the first equality says $\operatorname{Re}(c)\left\| \mathbf{v} \right\|^2 = \lvert c \rvert \left\| \mathbf{v} \right\|^2$ , so $\operatorname{Re}c = \lvert c \rvert$ , which forces $c$ to be real and $\ge 0$ .
(b) Apply Theorem 11.1.5 on $\mathbb{R}^n$ as in Exercise (B3: Cauchy–Schwarz, concretely) (b). Squaring removes the sign, so equality in the squared inequality is equality in $\lvert\langle \mathbf{a}, \mathbf{b} \rangle\rvert = \left\| \mathbf{a} \right\|\left\| \mathbf{b} \right\|$ , which by Theorem 11.1.5 holds exactly when $(\mathbf{a}, \mathbf{b})$ is dependent, that is, when one of the two vectors is a multiple of the other.
(c) Take $n = 1$ , $a_1 = 1$ , $b_1 = -1$ : then $\sum a_ib_i = -1$ and both sides of the classical inequality equal $1$ . There is no contradiction with (a), because (a) is about equality in Corollary 11.1.6, which needs a non-negative multiple, while the classical inequality is the squared Cauchy–Schwarz inequality, which is blind to the sign. Here $\left\| \mathbf{a} + \mathbf{b} \right\| = 0 \ne 2 = \left\| \mathbf{a} \right\| + \left\| \mathbf{b} \right\|$ .