*Least squares*. Let $V$ be a subspace of $\mathcal H$ and let
$f\in \mathcal H$. A vector $p \in V$ minimizes $\|f-v\|$ over all
$v\in V$ if and only if $f-p$ is orthogonal to $V$. If $p$ exists, it
is unique. Moreover, if $p$ exists for every $f\in \mathcal H$, then
there is an orthogonal projection $P_V$ such that $P_Vf=p$. Finally,
if $V$ is finite dimensional, the problem always has a solution, and
so $P_V$ always exists in that case.

When $V$ is finite dimensional, it has an o.n. basis $\{u_1,\ldots,u_n\}$ relative to which $P_V$ has the representation $P_Vf = \sum_{j=1}^n\langle f,u_j\rangle u_j$. Because $f-P_Vf$ is orthogonal to $V$, the Pythagorean theorem (or a simple computation) shows that \[ (\ast) \quad \| f - \sum_{j=1}^n\langle f,u_j\rangle u_j\|^2 = \|f-P_Vf\|^2 = \|f\|^2 - \underbrace{\sum_{j=1}^n \big|\langle f,u_j\rangle \big|^2}_{\|P_Vf\|^2}. \] We also will need to compute $\| f - \sum_{j=1}^n \alpha_j u_j\|^2$, for arbitrary $\alpha_j$'s. To do this, let $v=\sum_{j=1}^n \alpha_j u_j$. From this, we see that $f-v = f-P_Vf +P_Vf - v$. Since $f-P_Vf$ is orthogonal to $P_Vf - v \in V$, we can again use the Pythagorean theorem to obtain $\|f - v\|^2 = \|f - P_Vf\|^2 + \|v-P_Vf\|^2$. Putting this in terns of the basis then yields \[ (\ast \ast) \quad \| f - \sum_{j=1}^n \alpha_j u_j\|^2 = \|f\|^2 - \sum_{j=1}^n\big |\langle f,u_j\rangle \big|^2 + \sum_{j=1}^n\big |\alpha_j-\langle f,u_j\rangle \big|^2, \] which very clearly shows that the minimum occurs if and only if $\alpha_j = \langle f,u_j\rangle$.

*Complete orthonormal sets*. The question that now comes up is:
What happens when the o.n. set is infinite? Can we actually represent
every $f\in \mathcal H$ as $f = \sum_{j=1}^\infty\langle f,u_j\rangle
u_j$, where the series converges in $\mathcal H$? The answer is
``sometimes.'' When it is possible, we say that the o.n. set
$U:=\{u_j\}_{j=1}^\infty$ is *complete*. (Don't confuse this
completeness with completeness for a normed space. They are different
things.)

There are two equivalent definitions of completeness for an o.n. set:

- Every vector in $\mathcal H$ may be uniquely represented as the series $f=\sum_{j=1}^\infty \langle f, u_j\rangle u_j$.
- $U$ is maximal in the sense that there is no non-zero vector in $\mathcal H$ that is orthogonal to $U$. (Equivalently, $U$ is not a proper subset of any other o.n. set in $\mathcal H$.)

*Bessel's inequality *. The equation $(\ast)$ applies if the
space for $V=\text{span}\{u_j\}_{j=1}^n$. Since $\|f-P_Vf\|^2\ge 0$,
$(\ast)$ implies that $\|f\|^2 - \sum_{j=1}^n\big |\langle
f,u_j\rangle \big|^2 \ge 0$. Consequently, for every $n$ we have
$\sum_{j=1}^n\big |\langle f,u_j\rangle \big|^2 \le \|f\|^2$. Recall
that if the partial sums of a series with non negative terms are
uniformly bounded, then that series is convergent. This implies the
inequality below holds for all $f\in \mathcal H$. \[ (\text{Bessel's
inequality})\quad \sum_{j=1}^\infty \big |\langle f,u_j\rangle \big|^2
\le \|f\|^2. \]

*Parseval's equation*. We now come to our first necessary and
sufficient condition for completeness.

**Theorem**. An o.n. set $U$ is complete if and only if
\[
(\text{Parseval's equation}) \quad \|f\|^2 = \sum_{j=1}^\infty \big
|\langle f,u_j\rangle \big|^2 \ \text{holds for all} \ f\in \mathcal H.
\]

**Proof**. If $U$ is complete, then every $f\in \mathcal H$ is
represented by the series $f=\sum_{j=1}^\infty \langle f, u_j\rangle
u_j$, which converges in $\mathcal H$. That is, we have $\lim_{n\to
\infty} \|f-\sum_{j=1}^n \langle f, u_j\rangle u_j\|=0$. Using this in
$(\ast)$ then immediately implies that Parseval's equation holds for
all $f\in \mathcal H$. Conversely, if Parseval's equation holds for
all $f$ in $\mathcal H$ then $(\ast)$ implies that
\[
\lim_{n\to \infty} \|f-\sum_{j=1}^n \langle f, u_j\rangle u_j\|^2
= \|f\|^2 - \sum_{j=1}^\infty \big |\langle f,u_j\rangle \big|^2 =0,
\]
so $f = \sum_{j=1}^\infty \langle f, u_j\rangle u_j$. Consequently,
$U$ is complete.

*Subspace generated by an o.n. set*. Whether or not $U$ is
complete, we can still look at the subspace $\mathcal H_U : = \{g\in
\mathcal H : g = \sum_{j=1}^\infty \alpha_j u_j\}$. There are some
simple, but important consequences of $g$ being in $\mathcal H_U$. To
begin with, just using a simple limiting argument shows that $\langle
g,u_j\rangle = \alpha_j$. Put another way, the $\alpha_j$'s are
uniquely given by the previous formula.

The next thing involves $\mathcal H_U$ as a subspace of $\mathcal H$. Recall that a subset of $\mathcal H$ is closed if it contains all of its limit points. We want to prove the following result:

**Proposition**. $\mathcal H_U$ is a closed subspace of $\mathcal H$.

**Proof**. Let $\{g_\ell\}$ be a sequence in $\mathcal H_U$
convergent to $g\in \mathcal H$. We want to show that $g \in \mathcal
H_U$. To simplify the notation, we will let $P_n$ be the orthogonal
projection for $\text{span}\{u_j\}_{j=1}^n$. This allows us to write
$P_nf=\sum_{j=1}^n \langle f, u_j\rangle u_j$, so that we don't have
to use the sum in every case. Thus, our objective is to show that
$g=\lim_{n\to \infty}P_ng$. Indeed, $g\in \mathcal H_U$ if and only
$g=\lim_{n\to \infty}P_ng$.

First of all, $P_n g_\ell =\sum_{j=1}^n \langle g_\ell, u_j\rangle u_j$ is in the span of the first $n$ of the $u_j$'s. From the minimization properties of the projection (or, equivalently equation $(\ast \ast)$), we have that \[ \|g -P_ng\|\le \|g - P_ng_\ell\| \le \|g-g_\ell\| + \|g_\ell - P_n g_\ell\|. \] Now we employ the standard "up, over, and around" argument. For $\epsilon>0$, we can find $L$ such that $\|g-g_\ell\|<\frac12 \epsilon$ for all $\ell\ge L$. Now, choose $\ell\ge L$ and fix it. Because $g_\ell \in \mathcal H_U$, we also have that $g_\ell = \lim_{n\to \infty}P_n g_\ell$. Thus, there is an $N$ such that for $n \ge N$, $\|g_\ell - P_n g_\ell\| < \frac12 \epsilon$. From the inequality above, we thus get that $\|g -P_ng\| <\epsilon$ for all $n\ge N$. Thus $g=\lim_{n \to \infty}P_n g\in \mathcal H_U$ and, therefore, $\mathcal H_U$ is closed.

The following corollary immediately follows from the proposition above and the definition of completeness for an o.n. set.

**Corollary**. An o.n. set $U$ is complete if and only if $\mathcal
H_U = \mathcal H$.

*Checking for completeness*. In all of the results above,
checking whether an o.n. set is or is not complete involves the whole
space $\mathcal H$. Usually this is difficult to do. We now present an
alternative method that allows dealing only with a dense subset of
$\mathcal H$.

**Proposition**. Let $D$ be a dense subset of $\mathcal H$. An
o.n. set $U$ is complete if and only if $D\subset \mathcal H_U$.

**Proof**. Obviously, if $U$ is complete, $\mathcal H_U = \mathcal
H \supset D$. To show the converse, suppose that $D \subset
H_U$. Since $H_U$ is closed, it contains all of its limit points. But
$D$ is dense in $\mathcal H$, so its limit points comprise all of
$\mathcal H$. It follows that $\mathcal H_U= \mathcal H$ and that $U$
is complete.

We can restate the proposition above in an equivalent way, which will provide a direct check on showing the completeness of $U$.

**Corollary** Let $D$ be a dense subset of $\mathcal H$. An
o.n. set $U$ is complete if and only if every vector $f\in D$ can be
represented a series of the form $f=\sum_{j=1}^\infty \alpha_j u_j$.

**Proof**. The condition on $f\in D$ is simply a statement that
$f\in \mathcal H_U$, so the result follows from the previous
proposition.

**Proposition**. The polynomials are dense in $L^2[a,b]$.

**Proof**. Let $f\in L^2$. Because the continuous functions are
dense in $L^2$, for every $\epsilon >0$ we have that there is a $g\in
C[a,b]$ such that $\|f-g\|_{L^2} < \frac12 \epsilon$. By the
Weierstrass approximation theorem, which we will discuss next, we can
also find a polynomial $p$ such that $\|g - p\|_{C[a,b]}
<\frac{1}{2\sqrt{b-a}} \epsilon$. Note that for any $h \in C[a,b]$, we
have
\[ \| h \|_{L^2} = \bigg(\int_a^b |h(x)|^2dx\bigg)^{1/2}\le
(b-a)^{1/2} \|h\|_{C[a,b]}. \]
We thus have $\|f - p \|_{L^2}\le \|f-g\|_{L^2}+\|g-p\|_{L^2}\le
\|f-g\|_{L^2}+(b-a)^{1/2}\|g-p\|_{C[a,b]} <\frac12 \epsilon + \frac12
\epsilon=\epsilon$. Thus, the polynomials are dense in $L^2[a,b]$.

We point out that most sets of orthogonal polynomials are obtained by
using the Gram-Schmidt process on the set $\{1,x,x^2, \ldots\}$
relative to some *weighted* inner product, $\langle f,g\rangle
= \int_a^b f(x)\overline{g(x)}w(x)dx$. In the case where $w(x)=1$, the
result above proves that the polynomials orthogonal relative to the
inner product $\langle f,g\rangle = \int_a^b f(x)\overline{g(x)}dx$
form a complete set.

Let $a=-1$ and $b=1$, so that $\langle f,g\rangle = \int_{-1}^1
f(x)\overline{g(x)}dx$. The polynomials generated from this inner
product via the Gram-Schmidt process are called the *Legendre
polynomials*. The $\ell^{th}$ degree Legendre polynomial is
usually denoted by $P_\ell(x)$ and is normalized so that
$P_\ell(1)=1$; for example, we have
\[
P_0(x)=1,\ P_1(x) = x,\ P_2(x) =
\tfrac12\!(3x^2-1),\ P_3(x)=\tfrac12 \!(5x^3-3x).
\]
The $P_\ell$'s are orthogonal, but not orthonormal. It turns out that
$\|P_\ell\|_{L^2[-1,1]} = \sqrt{\frac{2}{2\ell+1}}$. Letting $p_\ell =
\sqrt{\frac{2\ell+1}{2}}P_\ell$ results in an orthonormal set,
$\{p_0,p_1, p_2\ldots\}$. For $\ell= 0$ to $\ell=4$, they are
\[
p_0(x)=\sqrt{\tfrac12},\ p_1(x) = \sqrt{\tfrac32} x,\ p_2(x)
=\sqrt{\tfrac58}(3x^2-1),\ p_3(x)=\sqrt{\tfrac78}(5x^3-3x).
\]
We can turn this around and write powers of $x$ in terms of the
$p_\ell $'s. For example,
\[
1=\sqrt{2}p_0,\ x = \sqrt{\tfrac23}\,p_1, \ x^2 =
\sqrt{\tfrac{2}{9}}p_0+ \sqrt{\tfrac{8}{45}}p_2,\ x^3 =
\sqrt{\tfrac{6}{25}}p_1+ \sqrt{\tfrac{8}{175}}p_3.
\]
The point is that every polynomial can be written as a linear
combination of the $p_\ell$'s, so they all belong to $\mathcal H_U$,
with $U=\{p_\ell\}_{\ell=0}^\infty$. By the two previous propositions,
we obtain this:

**Corollary.** The Legendre polynomials form a complete set in $L^2[-1,1]$.

Previous: Lebesgue integration

Next: Approximation of continuous functions

Updated 9/28/2015.