Inner Product Spaces

Nine chapters have described vector spaces without measuring anything in them. We can say that a list spans, that a subspace has dimension four, that two matrices are similar; we cannot say that a vector has length $3$, that two vectors are perpendicular, or that one vector is the closest point of a plane to another. Every picture in the book so far has secretly used lengths and angles that the theory did not supply. This chapter supplies them, with a single piece of extra structure: an inner product, abstracted from the dot product on ${\mathbb{R}}^n$ by keeping exactly the properties that make the law of cosines work.

The gain is out of all proportion to the cost. Length gives the Cauchy–Schwarz and triangle inequalities. Perpendicularity gives orthonormal bases, in which coordinates are read off by a single inner product and Gram–Schmidt produces one from any basis at all. Splitting a space into a subspace and its orthogonal complement gives the nearest point of a subspace to a given vector, and with it least squares, the standard answer to an inconsistent system. The Riesz representation theorem pays off a debt from Chapter 5: an inner product is precisely the extra data that makes $V$ and $V^{\ast }$ the same space. From Riesz comes the adjoint $T^{\ast }$, which turns the four fundamental subspaces into two orthogonal decompositions, and the isometries, which are the maps that preserve the whole geometry. The chapter ends with the computational and classical faces of the same ideas: the QR factorization, Fourier series and orthogonal polynomials. Throughout, the field is $\mathbb{R}$ or $\mathbb{C}$, because the construction needs both an ordering, so that a length squared can be positive, and complex conjugation, so that it can be real.

What you need. From Chapter 0, the complex conjugate and modulus with their arithmetic (Definition 0.6.2, Theorem 0.6.2) and the trace (Definition 0.9.12, Theorem 0.9.6). From Chapter 1, independence, bases, dimension, basis extension and direct sums (Definition 1.4.1, Theorem 1.6.4, Theorem 1.6.5, Theorem 1.7.4). From Chapter 2, the coordinate isomorphism and Rank–Nullity (Corollary 2.5.5, Theorem 2.3.1). From Chapter 3, the Invertible Matrix Theorem (Theorem 3.4.4). From Chapter 4, the matrix of a linear map (Definition 4.1.1). From Chapter 5, the dual space, the annihilator and the dual map, together with the warning that an isomorphism $V\cong V^{\ast }$ needs a choice (Definition 5.1.2, Definition 5.2.1, Definition 5.4.1, Theorem 5.4.2). From Chapter 6, the root bound and Lagrange interpolation (Corollary 6.1.4, Theorem 6.6.1). From Chapter 7, multiplicativity of the determinant and the non-negativity of Gram determinants (Theorem 7.5.4, Corollary 7.7.3). From Chapter 8 §01, one fact, used four times in §08: a product of partitioned matrices multiplies blockwise (Theorem 8.1.1). Chapter 9 is used only in examples and in one classification (Definition 9.2.1); nothing before Section 7 needs an eigenvalue.

Roadmap

• Inner products and norms. The axioms, abstracted from the dot product; length, orthogonality, Pythagoras, Cauchy–Schwarz and the triangle inequality.
• Orthonormal bases and Gram–Schmidt. Coordinates that cost one inner product each, the process that builds such a basis from any basis, and Gram matrices.
• Orthogonal complements and projections. The splitting $V=U\oplus U^{\perp }$, the projection $P_U$, and the fact that $P_U\mathbf{v}$ is the point of $U$ nearest to $\mathbf{v}$.
• Least squares and minimum-norm solutions. What to do with an inconsistent system: the normal equations, and the smallest solution of an underdetermined one.
• The Riesz representation theorem. Every linear functional is an inner product with one fixed vector, which is the chosen isomorphism $V\cong V^{\ast }$ that Chapter 5 could not supply.
• The adjoint. Moving an operator to the other side of an inner product, the matrix ${\mathbf{A}}^{\ast }$, and the four fundamental subspaces in orthogonal form.
• Isometries and unitary maps. The maps that preserve all lengths, their many equivalent descriptions, the Frobenius norm and its blindness to unitary factors, and the classification of $\mathrm{O}(2)$.
• QR, Householder and Hessenberg. Gram–Schmidt as a matrix factorization, reflections that do the same job more robustly, and the first step of a serious eigenvalue algorithm.
• Bessel, Parseval and Fourier series. What an orthonormal list captures of a vector, and the classical Fourier coefficients as an instance.
• Orthogonal polynomials. Gram–Schmidt on $1,x,x^2,\dots$, the three-term recurrence, real roots, and Gaussian quadrature.

Named moves

• To show a vector is zero, pair it with itself. Only $\mathbf{0}$ has $⟨\mathbf{z},\mathbf{z}⟩=0$, so an identity of that form ends the argument.
• To move $T$ across an inner product, pay with $T^{\ast }$. The adjoint is the price of the transfer, and most proofs about special operators compute one inner product in two ways.
• Work with the norm squared, then expand. Square roots do not expand; ${\Vert \mathbf{u}+\mathbf{v}\Vert }^2=⟨\mathbf{u}+\mathbf{v},\mathbf{u}+\mathbf{v}⟩$ does.
• Fix an orthonormal basis. In its coordinates every inner product becomes the dot product, and an abstract statement becomes a matrix statement.
• Project, then subtract. The same move orthogonalizes a list (Gram–Schmidt), finds the nearest point of a subspace, and solves a least-squares problem.