Minimum Residual Energy

In the notes on signal spaces we commented on interpreting the integral $\int_{-\pi}^\pi|f(t)|^2dt$ as the energy in a signal. The question that we want to answer here is what "trig polynomial" $T_N(t)=A_0+\sum_{k=1}^N A_k \cos(kt)+B_k\sin(kt)$ will give the best approximation to $f$, in the sense of minimizing the residual energy, \[ (\dagger) \quad \int_{-\pi}^\pi(f(t)-T_N(t))^2dt. \] We will begin by showing that functions for that \[ (\ast) \quad \int_{-\pi}^\pi f(t)T_N(t)dt = 2\pi a_0A_0 + \pi \sum_{k=1}^N a_k A_k+b_k B_k. \] To see this, note that \[ \begin{align} \int_{-\pi}^\pi f(t)T_N(t)dt =& \int_{-\pi}^\pi f(t) \bigg(A_0+\sum_{k=1}^N A_k \cos(kt)+B_k\sin(kt)\bigg)dt \\ =& A_0\int_{-\pi}^\pi f(t)dt + \sum_{k=1}^N A_k \int_{-\pi}^\pi f(t)\cos(kt)dt + B_k \int_{-\pi}^\pi f(t)\sin(kt)dt. \end{align} \] Recall that $a_0=\frac{1}{2\pi}\int_{-\pi}^\pi f(t)dt$, and for $k\ge 1$, $a_k = \frac{1}{\pi}\int_{-\pi}^\pi f(t)\cos(kt)dt$. Similarly, $b_k = \frac{1}{\pi}\int_{-\pi}^\pi f(t)\sin(kt)dt$. Using these in the previous equation results in $(\ast)$. Since $f$ is arbitrary, we may replace it by $T_N$. Doing so in $(\ast)$ results in the formulas below: \[ (\ast \ast)\quad \int_{-\pi}^\pi T_N(t)^2dt =2\pi A_0^2+ \pi \sum_{k=1}^N A_k^2+B_k^2. \] We can now compute the residual energy defined in $(\dagger)$. \[ \begin{align} \int_{-\pi}^\pi(f(t)-T_N(t))^2dt =& \int_{-\pi}^\pi f(t)^2dt - 2\int_{-\pi}^\pi f(t)T_N(t)dt + \int_{-\pi}^\pi T_N(t))^2dt \\ =& \int_{-\pi}^\pi f(t)^2dt -4\pi a_0A_0 - 2\pi \sum_{k=1}^N \big(a_k A_k+b_k B_k\big) + 2\pi A_0^2+ \pi \sum_{k=1}^N\big( A_k^2+B_k^2)\\. \end{align} \] Next, complete the square for the terms on the right in the last line above: $2\pi(A_0^2 -2a_0 A_0) = 2\pi (A_0-a_0)^2 - 2\pi a_0^2$, $\pi (A_k^2 -A_k) = \pi (A_k-a_k)^2 - \pi a_k^2$, and finally $\pi (B_k^2 -b_k) = \pi (B_k-b_k)^2 - \pi b_k^2$. The result is \[ \int_{-\pi}^\pi(f(t)-T_N(t))^2dt = \int_{-\pi}^\pi f(t)^2dt + 2\pi (A_0^2-a_0)^2+ \pi \sum_{k=1}^N \big (A_k-a_k)^2 + \pi (B_k-b_k)^2\big) - 2\pi a_0^2 -\pi \sum_{k=1}^N (a_k^2+b_k^2) \] The middle term will be strictly positive if $A_k\ne a_k$ and $B_k\ne b_k$, even for one $k$; consequently, if $T_N(t)\ne S_N(t)$, \[ \int_{-\pi}^\pi(f(t)-T_N(t))^2dt > \int_{-\pi}^\pi f(t)^2dt -2\pi a_0^2 -\pi \sum_{k=1}^N (a_k^2+b_k^2)= \int_{-\pi}^\pi (f(t)-S_N(t))^2dt \] Thus the minimum occurs exactly when $T_N(t)=S_N(t)$. Moreover, the minimizer is unique; that is,only the partial sum $S_N$ is the minimizer. Using the two terms on the right above, we see that the minimum residual energy is given by \[ (\ddagger)\quad \int_{-\pi}^\pi(f(t)-S_N(t))^2dt = \int_{-\pi}^\pi f(t)^2dt -2\pi a_0^2 -\pi \sum_{k=1}^N (a_k^2+b_k^2) = \int_{-\pi}^\pi f(t)^2dt - \int_{-\pi}^\pi S_N(t)^2dt\]

Convergence in the mean

We say that the partial sum $S_N$ converges to $f$ in the mean if and only if $\lim_{N\to \infty}\int_{-\pi}^\pi(F(t)- S_N(t)^2dt =0$. Using the expression for the minimal energy in $(\ddagger)$ we have the following proposition.

Theorem (Parseval) The partial sum $S_N$ converges in the mean to $f$ if and only if Parseval's equation holds: \[ (\S) \quad \int_{-\pi}^\pi f(t)^2dt = 2\pi a_0^2 +\pi \sum_{k=1}^\infty (a_k^2+b_k^2) \]

Proof By $(\ddagger)$, if $S_N$ converges in the mean to $f$, then \[ 0=\lim_{N\to \infty} \int_{-\pi}^\pi(f(t)-S_N(t))^2dt =\int_{-\pi}^\pi f(t)^2dt -2\pi a_0^2 -\pi \lim_{N\to \infty}\sum_{k=1}^N (a_k^2+b_k^2), \] so Parseval's equation holds. Conversely, if $2\pi a_0^2 +\pi \sum_{k=1}^\infty (a_k^2+b_k^2) = \int_{-\pi}^\pi f(t)^2dt$, then, again by $(\ddagger)$, $S_N$ converges to $f$ in the mean.

Theorem (Riesz-Fischer) Suppose that $f$ is in $L^2[-\pi,\pi]$. The $S_N$ converges in the mean to $f$. Conversely, suppose that the sequence $\{a_0, a_1, b_1, a_2,b_2, \ldots a_k,b_k \dots\} $ satisfies $2\pi a_0^2 +\pi \sum_{k=1}^\infty (a_k^2+b_k^2)<\infty$, then there exists a unique $f$ in $L^2[-\pi,\pi]$ such that $S_N(t)= a_0+\sum_{k=1}^N a_k \cos(kt)+b_k\sin(kt)$ converges to $f$ in the mean.

The proof of this theorem is beyond the scope of this course. To describe what it says we need to know the meaning of Parseval's equation $(\S)$. Remember that the left side is interpreted as the energy in the signal $f$. But what about the right side? If consider a single term of the form $a_k \cos(kt)$, then the energy in this term $2\pi a_0^2$, for $k=0$, and $\pi a_k^2$ for $k\ge 1$. The same is true for the $b_k$'s. The energy in $b_k\sin(kt)$ is $\pi b_k^2$. The point is that Parseval's equation $(\S)$ represents conservation of energy. The energy in the signal is the sum of the energies in the individual frequencies.

The Riesz-Fischer Theorem has an interesting interpretation. The first is that for any finite energy signal, the energy in the sum of its frequencies components equals the energy in the signal, as mentioned above. But the reverse is also true. Given an sequence of $a$'s and $b$'s such that the sum of the energies in the individual frequencies is finite, then there is a unique signal $f$, with finite energy, having the $a$'s and $b$'s from the sequence as its Fourier series.

Let's think of this in another way. Each of the terms $a_k\cos(kt)$ or $ b_k\sin(kt)$ represents a signal with pure (angular) frequency $k$. The amplitude of each of these, $|a_k|$ or $|b_k|$ represents the intensity of a frequency — for example, how loud the sound corresponding to the frequency component is. What the Riesz-Fischer Theorem implies is that for any sequence of $a$'s and $b$'s, as long as the sum $2\pi a_0^2 +\pi \sum_{k=1}^\infty (a_k^2+b_k^2)<\infty$, there will be a corresponding finite energy signal. Put another way, you can synthesize a signal with arbitrary frequencies in it, provided the total energy in the components is finite.