We begin with Green's Theorem in "flux" or "divergence" form: \[ \oint_C \mathbf F\cdot \mathbf n ds = \int\!\int_\Omega \nabla\cdot \mathbf F dA, \] where $\mathbf F$ is a vector field that is continuously differentiable in $\Omega$. Let $u$ be the solution to the eigenvalue problem and take $\mathbf F = u\nabla u$. With this choice for $\mathbf F$, Green's Theorem implies that \[ \oint_C u \nabla u\cdot \mathbf n ds = \int\!\int_\Omega \nabla\cdot (u\nabla u) dA. \] Two things. First, since $u|_C=0$, we have $\oint_C u_C\nabla u \cdot \mathbf n ds = \oint_C 0 \nabla u\cdot \mathbf n ds = 0 = \int\!\int_\Omega \nabla\cdot (u\nabla u)dA$. Second, $\nabla\cdot (u\nabla u) = \nabla u \cdot \nabla u +u\nabla^2 u$. Putting these two together gives us \[ 0=\int\!\int_\Omega \nabla\cdot (u\nabla u)dA = \int\!\int_\Omega u\nabla^2 u dA + \int\!\int_\Omega |\nabla u|^2dA. \] Consequently, we have \[ (\ast) \quad \int\!\int_\Omega u\nabla^2 u dA = - \int\!\int_\Omega |\nabla u|^2dA. \]
To get the Rayleigh quotient, follow these steps.