Least Squares and Fourier Series
We want to find the minimum of $\int_{-\pi}^\pi|f(x)-T_N(x)|^2dx$,
where $T_N\in \text{span}\{e^{inx}\}_{n=-N}^N\ $ i.e.,
$T_N(x)=\sum_{n=-N}^N \beta_n e^{inx}$. In class, we used the
technique from section 0.5 in the text. Here, we will use a more
direct method. First of all,
\[
\int_{-\pi}^\pi|f(x)-T_N(x)|^2dx = \int_{-\pi}^\pi|f(x)|^2dx
-\int_{-\pi}^\pi f(x)\overline{T_N(x)}dx -\int_{-\pi}^\pi
\overline{f(x)}T_N(x)dx + \int_{-N}^N |T_N(x)|^2dx
\]
Let's compute $\int_{-\pi}^\pi f(x)\overline{T_N(x)}dx$. Put the
expression for $T_N$ into the integral and use a little calculus to
get
\[ \int_{-\pi}^\pi f(x)\overline{T_N(x)}dx = \sum_{n=-N}^N
\int_{-\pi}^\pi f(x)\overline{\beta_n e^{inx}}dx=\sum_{n=-N}^N
\overline{\beta_n} \underbrace{\int_{-\pi}^\pi f(x)e^{-inx}}_{2\pi
\alpha_n} = 2\pi \sum_{n=-N}^N
\overline{\beta_n}\alpha_n\]
Because $\overline{\int_{-\pi}^\pi
\overline{f(x)}T_N(x)dx}=\int_{-\pi}^\pi f(x)\overline{T_N(x)}dx$, we have that
\[ \int_{-\pi}^\pi \overline{f(x)}T_N(x)dx = 2\pi\sum_{n=-N}^N
\overline{\overline{\beta_n}\alpha_n} =2\pi
\sum_{n=-N}^N\overline{\alpha_n}\beta_n \]
Using a similar calculation (Can you do it?), one can also show that
$\int_{-N}^N |T_N(x)|^2dx =2\pi \sum_{n=-N}^N |\beta_n|^2$. From this
and the two previous formulas we see that
\[ \int_{-\pi}^\pi|f(x)-T_N(x)|^2dx = \int_{-\pi}^\pi|f(x)|^2dx
-2\pi\sum_{n=-N}^N \left(\overline{\beta_n}\alpha_n +
\overline{\alpha_n}\beta_n\right) + 2\pi\sum_{n=-N}^N |\beta_n|^2 =
\int_{-\pi}^\pi|f(x)|^2dx+2\pi\sum_{n=-N}^N\left(
|\beta_n|^2-\overline{\beta_n}\alpha_n -
\overline{\alpha_n}\beta_n\right) \]
By completing the square, we have
$|\beta_n|^2-\overline{\beta_n}\alpha_n - \overline{\alpha_n}\beta_n=
|\beta_n -\alpha_n|^2-|\alpha_n|^2$. It follows that
\[ \int_{-\pi}^\pi|f(x)-T_N(x)|^2dx = \int_{-\pi}^\pi|f(x)|^2dx
+2\pi\sum_{n=-N}^N |\beta_n-\alpha_n|^2 - 2\pi\sum_{n=-N}^N
|\alpha_n|^2. \]
The middle term will be strictly positive if $\beta_n\ne\alpha_n$ even
for one $n$; consequently, if $T_N(x)\ne S_N(x)$
\[ \int_{-\pi}^\pi|f(x)-T_N(x)|^2dx > \int_{-\pi}^\pi|f(x)|^2dx -
2\pi\sum_{n=-N}^N |\alpha_n|^2 = \int_{-\pi}^\pi|f(x)-S_N(x)|^2dx \]
Thus the minimum occurs exactly when $T_N(x)=S_N(x)$. Moreover, the
minimiizer is unique; that is,only the partial sum is the
minimizer. Finally, we write down the minimum itself:
\[ \int_{-\pi}^\pi|f(x)-S_N(x)|^2dx = \int_{-\pi}^\pi|f(x)|^2dx -
2\pi\sum_{n=-N}^N |\alpha_n|^2. \]