Least squares. Let $V$ be a subspace of $\mathcal H$ and let $f\in \mathcal H$. A vector $p \in V$ minimizes $\|f-v\|$ over all $v\in V$ if and only if $f-p$ is orthogonal to $V$. If $p$ exists, it is unique. Moreover, if $p$ exists for every $f\in \mathcal H$, then there is an orthogonal projection $P_V$ such that $P_Vf=p$. Finally, if $V$ is finite dimensional, the problem always has a solution, and so $P_V$ always exists in that case.
When $V$ is finite dimensional, it has an o.n. basis $\{u_1,\ldots,u_n\}$ relative to which $P_V$ has the representation $P_Vf = \sum_{j=1}^n\langle f,u_j\rangle u_j$. Because $f-P_Vf$ is orthogonal to $V$, the Pythagorean theorem (or a simple computation) shows that \[ (\ast) \quad \| f - \sum_{j=1}^n\langle f,u_j\rangle u_j\|^2 = \|f-P_Vf\|^2 = \|f\|^2 - \underbrace{\sum_{j=1}^n \big|\langle f,u_j\rangle \big|^2}_{\|P_Vf\|^2}. \] We also will need to compute $\| f - \sum_{j=1}^n \alpha_j u_j\|^2$, for arbitrary $\alpha_j$'s. To do this, let $v=\sum_{j=1}^n \alpha_j u_j$. From this, we see that $f-v = f-P_Vf +P_Vf - v$. Since $f-P_Vf$ is orthogonal to $P_Vf - v \in V$, we can again use the Pythagorean theorem to obtain $\|f - v\|^2 = \|f - P_Vf\|^2 + \|v-P_Vf\|^2$. Putting this in terns of the basis then yields \[ (\ast \ast) \quad \| f - \sum_{j=1}^n \alpha_j u_j\|^2 = \|f\|^2 - \sum_{j=1}^n\big |\langle f,u_j\rangle \big|^2 + \sum_{j=1}^n\big |\alpha_j-\langle f,u_j\rangle \big|^2, \] which very clearly shows that the minimum occurs if and only if $\alpha_j = \langle f,u_j\rangle$.
Complete orthonormal sets. The question that now comes up is: What happens when the o.n. set is infinite? Can we actually represent every $f\in \mathcal H$ as $f = \sum_{j=1}^\infty\langle f,u_j\rangle u_j$, where the series converges in $\mathcal H$? The answer is ``sometimes.'' When it is possible, we say that the o.n. set $U:=\{u_j\}_{j=1}^\infty$ is complete. (Don't confuse this completeness with completeness for a normed space. They are different things.)
There are two equivalent definitions of completeness for an o.n. set:
Bessel's inequality . The equation $(\ast)$ applies if the space for $V=\text{span}\{u_j\}_{j=1}^n$. Since $\|f-P_Vf\|^2\ge 0$, $(\ast)$ implies that $\|f\|^2 - \sum_{j=1}^n\big |\langle f,u_j\rangle \big|^2 \ge 0$. Consequently, for every $n$ we have $\sum_{j=1}^n\big |\langle f,u_j\rangle \big|^2 \le \|f\|^2$. Recall that if the partial sums of a series with non negative terms are uniformly bounded, then that series is convergent. This implies the inequality below holds for all $f\in \mathcal H$. \[ (\text{Bessel's inequality})\quad \sum_{j=1}^\infty \big |\langle f,u_j\rangle \big|^2 \le \|f\|^2. \]
Parseval's equation. We now come to our first necessary and sufficient condition for completeness.
Theorem. An o.n. set $U$ is complete if and only if \[ (\text{Parseval's equation}) \quad \|f\|^2 = \sum_{j=1}^\infty \big |\langle f,u_j\rangle \big|^2 \ \text{holds for all} \ f\in \mathcal H. \]
Proof. If $U$ is complete, then every $f\in \mathcal H$ is represented by the series $f=\sum_{j=1}^\infty \langle f, u_j\rangle u_j$, which converges in $\mathcal H$. That is, we have $\lim_{n\to \infty} \|f-\sum_{j=1}^n \langle f, u_j\rangle u_j\|=0$. Using this in $(\ast)$ then immediately implies that Parseval's equation holds for all $f\in \mathcal H$. Conversely, if Parseval's equation holds for all $f$ in $\mathcal H$ then $(\ast)$ implies that \[ \lim_{n\to \infty} \|f-\sum_{j=1}^n \langle f, u_j\rangle u_j\|^2 = \|f\|^2 - \sum_{j=1}^\infty \big |\langle f,u_j\rangle \big|^2 =0, \] so $f = \sum_{j=1}^\infty \langle f, u_j\rangle u_j$. Consequently, $U$ is complete.
Subspace generated by an o.n. set. Whether or not $U$ is complete, we can still look at the subspace $\mathcal H_U : = \{g\in \mathcal H : g = \sum_{j=1}^\infty \alpha_j u_j\}$. There are some simple, but important consequences of $g$ being in $\mathcal H_U$. To begin with, just using a simple limiting argument shows that $\langle g,u_j\rangle = \alpha_j$. Put another way, the $\alpha_j$'s are uniquely given by the previous formula.
The next thing involves $\mathcal H_U$ as a subspace of $\mathcal H$. Recall that a subset of $\mathcal H$ is closed if it contains all of its limit points. We want to prove the following result:
Proposition. $\mathcal H_U$ is a closed subspace of $\mathcal H$.
Proof. Let $\{g_\ell\}$ be a sequence in $\mathcal H_U$ convergent to $g\in \mathcal H$. We want to show that $g \in \mathcal H_U$. To simplify the notation, we will let $P_n$ be the orthogonal projection for $\text{span}\{u_j\}_{j=1}^n$. This allows us to write $P_nf=\sum_{j=1}^n \langle f, u_j\rangle u_j$, so that we don't have to use the sum in every case. Thus, our objective is to show that $g=\lim_{n\to \infty}P_ng$. Indeed, $g\in \mathcal H_U$ if and only $g=\lim_{n\to \infty}P_ng$.
First of all, $P_n g_\ell =\sum_{j=1}^n \langle g_\ell, u_j\rangle u_j$ is in the span of the first $n$ of the $u_j$'s. From the minimization properties of the projection (or, equivalently equation $(\ast \ast)$), we have that \[ \|g -P_ng\|\le \|g - P_ng_\ell\| \le \|g-g_\ell\| + \|g_\ell - P_n g_\ell\|. \] Now we employ the standard "up, over, and around" argument. For $\epsilon>0$, we can find $L$ such that $\|g-g_\ell\|<\frac12 \epsilon$ for all $\ell\ge L$. Now, choose $\ell\ge L$ and fix it. Because $g_\ell \in \mathcal H_U$, we also have that $g_\ell = \lim_{n\to \infty}P_n g_\ell$. Thus, there is an $N$ such that for $n \ge N$, $\|g_\ell - P_n g_\ell\| < \frac12 \epsilon$. From the inequality above, we thus get that $\|g -P_ng\| <\epsilon$ for all $n\ge N$. Thus $g=\lim_{n \to \infty}P_n g\in \mathcal H_U$ and, therefore, $\mathcal H_U$ is closed.
The following corollary immediately follows from the proposition above and the definition of completeness for an o.n. set.
Corollary. An o.n. set $U$ is complete if and only if $\mathcal H_U = \mathcal H$.
Checking for completeness. In all of the results above, checking whether an o.n. set is or is not complete involves the whole space $\mathcal H$. Usually this is difficult to do. We now present an alternative method that allows dealing only with a dense subset of $\mathcal H$.
Proposition. Let $D$ be a dense subset of $\mathcal H$. An o.n. set $U$ is complete if and only if $D\subset \mathcal H_U$.
Proof. Obviously, if $U$ is complete, $\mathcal H_U = \mathcal H \supset D$. To show the converse, suppose that $D \subset H_U$. Since $H_U$ is closed, it contains all of its limit points. But $D$ is dense in $\mathcal H$, so its limit points comprise all of $\mathcal H$. It follows that $\mathcal H_U= \mathcal H$ and that $U$ is complete.
We can restate the proposition above in an equivalent way, which will provide a direct check on showing the completeness of $U$.
Corollary Let $D$ be a dense subset of $\mathcal H$. An o.n. set $U$ is complete if and only if every vector $f\in D$ can be represented a series of the form $f=\sum_{j=1}^\infty \alpha_j u_j$.
Proof. The condition on $f\in D$ is simply a statement that $f\in \mathcal H_U$, so the result follows from the previous proposition.
Proposition. The polynomials are dense in $L^2[a,b]$.
Proof. Let $f\in L^2$. Because the continuous functions are dense in $L^2$, for every $\epsilon >0$ we have that there is a $g\in C[a,b]$ such that $\|f-g\|_{L^2} < \frac12 \epsilon$. By the Weierstrass approximation theorem, which we will discuss next, we can also find a polynomial $p$ such that $\|g - p\|_{C[a,b]} <\frac{1}{2\sqrt{b-a}} \epsilon$. Note that for any $h \in C[a,b]$, we have \[ \| h \|_{L^2} = \bigg(\int_a^b |h(x)|^2dx\bigg)^{1/2}\le (b-a)^{1/2} \|h\|_{C[a,b]}. \] We thus have $\|f - p \|_{L^2}\le \|f-g\|_{L^2}+\|g-p\|_{L^2}\le \|f-g\|_{L^2}+(b-a)^{1/2}\|g-p\|_{C[a,b]} <\frac12 \epsilon + \frac12 \epsilon=\epsilon$. Thus, the polynomials are dense in $L^2[a,b]$.
We point out that most sets of orthogonal polynomials are obtained by using the Gram-Schmidt process on the set $\{1,x,x^2, \ldots\}$ relative to some weighted inner product, $\langle f,g\rangle = \int_a^b f(x)\overline{g(x)}w(x)dx$. In the case where $w(x)=1$, the result above proves that the polynomials orthogonal relative to the inner product $\langle f,g\rangle = \int_a^b f(x)\overline{g(x)}dx$ form a complete set.
Let $a=-1$ and $b=1$, so that $\langle f,g\rangle = \int_{-1}^1 f(x)\overline{g(x)}dx$. The polynomials generated from this inner product via the Gram-Schmidt process are called the Legendre polynomials. The $\ell^{th}$ degree Legendre polynomial is usually denoted by $P_\ell(x)$ and is normalized so that $P_\ell(1)=1$; for example, we have \[ P_0(x)=1,\ P_1(x) = x,\ P_2(x) = \tfrac12\!(3x^2-1),\ P_3(x)=\tfrac12 \!(5x^3-3x). \] The $P_\ell$'s are orthogonal, but not orthonormal. It turns out that $\|P_\ell\|_{L^2[-1,1]} = \sqrt{\frac{2}{2\ell+1}}$. Letting $p_\ell = \sqrt{\frac{2\ell+1}{2}}P_\ell$ results in an orthonormal set, $\{p_0,p_1, p_2\ldots\}$. For $\ell= 0$ to $\ell=4$, they are \[ p_0(x)=\sqrt{\tfrac12},\ p_1(x) = \sqrt{\tfrac32} x,\ p_2(x) =\sqrt{\tfrac58}(3x^2-1),\ p_3(x)=\sqrt{\tfrac78}(5x^3-3x). \] We can turn this around and write powers of $x$ in terms of the $p_\ell $'s. For example, \[ 1=\sqrt{2}p_0,\ x = \sqrt{\tfrac23}\,p_1, \ x^2 = \sqrt{\tfrac{2}{9}}p_0+ \sqrt{\tfrac{8}{45}}p_2,\ x^3 = \sqrt{\tfrac{6}{25}}p_1+ \sqrt{\tfrac{8}{175}}p_3. \] The point is that every polynomial can be written as a linear combination of the $p_\ell$'s, so they all belong to $\mathcal H_U$, with $U=\{p_\ell\}_{\ell=0}^\infty$. By the two previous propositions, we obtain this:
Corollary. The Legendre polynomials form a complete set in $L^2[-1,1]$.
Previous: Lebesgue integration
Next: Approximation of continuous functions
Updated 9/28/2015.