Orthogonality of Associated Legendre Polynomials
Theorem
The associated Legendre polynomials over the interval $[-1,1]$ for a fixed $m$ form an orthogonal set.
$$ \int_{-1}^{1} P_{l}^{m}(x)P_{k}^{m}(x)dx =\frac{ 2}{ 2l+1 }\frac{(l+m)!}{(l-m)!}\delta_{lk} $$
In the case of $x=\cos \theta$,
$$ \int_{0}^{\pi} P_{l}^{m}(\cos \theta)P_ {k}^{m}(\cos\theta)\sin \theta d\theta =\frac{ 2}{ 2l+1 }\frac{(l+m)!}{(l-m)!}\delta_{lk} $$
Associated Legendre Polynomials $$ P_{l}^{m}(x) = (1-x ^{2})^{\frac{m}{2}} \dfrac{1}{2^l l!} \dfrac{d^{l+m}}{dx^{l+m}}(x^2-1)^l $$
Proof
For convenience, let’s briefly denote it as $P_{lm} = P_{l}^{m}(x)$. The associated Legendre differential equation is as follows.
$$ \begin{equation} \frac{d}{dx} \left[ (1-x^{2})P_{lm}^{\prime} \right] +\left[ l(l+1)-\frac{m^{2}}{1-x^{2}} \right]P_{lm} = 0 \end{equation} $$
Case 1: $l \ne k$
The proof method is the same as showing the orthogonality of Legendre polynomials. Writing $(1)$ for $l$ and $k$,
$$ \frac{d}{dx} \left[(1-x^{2}) P_{lm}^{\prime}\right] + \left[l(l+1) - \frac{m^{2}}{1-x^{2}}\right] P_{lm} = 0 \\ \frac{d}{dx} \left[(1-x^{2}) P_{km}^{\prime}\right] + \left[k(k+1) - \frac{m^{2}}{1-x^{2}}\right] P_{km} = 0 $$
Multiply $P_{km}$ by the equation for $l$, and $P_{lm}$ by the equation for $k$, and subtract one from the other to get the following.
$$ \begin{equation} P_{km} \frac{d}{dx} \left[ (1-x^{2})P_{lm}^{\prime} \right]-P_{lm}\frac{d}{dx} \left[ (1-x^{2})P_{km}^{\prime} \right]+\left[l(l+1)- k(k+1) \right]P_{lm}P_{km} = 0 \end{equation} $$
The first and second terms can be organized as follows.
$$ \begin{align*} &\quad \ P_{km} \frac{d}{dx} \left[ (1-x^{2})P_{lm}^{\prime} \right]-P_{lm}\frac{d}{dx} \left[ (1-x^{2})P_{km}^{\prime} \right] \\ &= P_{km}(1-x^{2})^{\prime}P_{lm}^{\prime}+P_{km}(1-x^{2})P_{lm}^{\prime \prime}-P_{lm}(1-x^{2})^{\prime}P_{km}^{\prime}-P_{lm}(1-x^{2})P_{km}^{\prime \prime} \\ &= {\color{blue}P_{km}(1-x^{2})^{\prime}P_{lm}^{\prime}+P_{km}(1-x^{2})P_{lm}^{\prime \prime}}-{\color{orange}P_{lm}(1-x^{2})^{\prime}P_{km}^{\prime}-P_{lm}(1-x^{2})P_{km}^{\prime \prime} } \\ &\quad + {\color{blue}P_{km}^{\prime}(1-x^{2})P_{lm}^{\prime}}-{\color{orange}P_{km}^{\prime}(1-x^{2})P_{lm}^{\prime}} \\ &= \frac{d}{dx}\left[{\color{blue}(1-x^{2})P_{lm}^{\prime}P_{km}}-{\color{orange}(1-x^{2})P_{lm}P_{km}^{\prime} } \right] \\ &= \frac{d}{dx}\left[(1-x^{2})(P_{lm}^{\prime}P_{km}-P_{lm}P_{km}^{\prime}) \right] \end{align*} $$
Inserting this into $(2)$,
$$ \frac{d}{dx}\left[(1-x^{2})(P_{lm}^{\prime}P_{km}-P_{lm}P_{km}^{\prime}) \right]+\left[l(l+1)- k(k+1) \right]P_{lm}P_{km}=0 $$
Integrating both sides over the interval $[-1,1]$ yields the following.
$$ \left[(1-x^{2})(P_{lm}^{\prime}P_{km}-P_{lm}P_{km}^{\prime}) \right]_{-1}^{1}+\left[l(l+1)- k(k+1) \right]\int_{-1}^{1}P_{lm}P_{km}dx=0 $$
The first term is $0$, so it goes as follows.
$$ \left[l(l+1)- k(k+1) \right]\int_{-1}^{1}P_{lm}P_{km}dx=0 $$
Since $l \ne k$, it results in $l(l+1)-k(k+1)\ne 0$. Thus, it follows that.
$$ \int_{-1}^{1}P_{l}^{m}(x)P_{k}^{m}(x)dx=0 $$
■
Case 2: $l=k$
The following holds.
$$ \frac{d^{l+m}}{dx^{l+m}}(x^{2}-1)^{l}=\frac{(l+m)!}{(l-m)!}(x^{2}-1)^{-m}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \tag {3} $$
By applying the above formula to the associated Legendre polynomials,
$$ \begin{align*} P_{l}^{m}(x) &= (1-x ^{2})^{\frac{m}{2}} \dfrac{1}{2^l l!} \dfrac{d^{l+m}}{dx^{l+m}}(x^2-1)^l \\ &= (1-x ^{2})^{\frac{m}{2}}\dfrac{1}{2^l l!} \frac{(l+m)!}{(l-m)!}(x^{2}-1)^{-m}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \\ &= \dfrac{(-1)^{m}}{2^l l!} \frac{(l+m)!}{(l-m)!}(1-x^{2})^{-\frac{m}{2}}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \end{align*} $$
Squaring both sides of the above equation yields the following.
$$ [P_{l}^{m}(x)]^{2} =\dfrac{1}{2^{2l}(l!)^{2}} \left[ \frac{(l+m)!}{(l-m)!} \right]^{2}(1-x^{2})^{-m}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} $$
By substituting $(3)$ into the above equation,
$$ \begin{align*} &\quad \ [P_{l}^{m}(x)]^{2} \\ &= \dfrac{1}{2^{2l}(l!)^{2}} \left[ \frac{(l+m)!}{(l-m)!} \right]^{2}(1-x^{2})^{-m}\left[ \frac{(l-m)!}{(l+m)!}(x^{2}-1)^{m}\frac{ d ^{l+m}}{ dx^{l+m} }(x^{2}-1)^{l} \right]\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \\ &= \dfrac{(-1)^{m}}{2^{2l}(l!)^{2}} \frac{(l+m)!}{(l-m)!} \frac{ d ^{l+m}}{ dx^{l+m} }(x^{2}-1)^{l}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \end{align*} $$
Now, integrating both sides over the interval $[-1,1]$ yields the following.
$$ \begin{align*} &\quad \ \int_{-1}^{1}[P_{l}^{m}(x)]^{2}dx \\ &= \dfrac{(-1)^{m}}{2^{2l}(l!)^{2}} \frac{(l+m)!}{(l-m)!}\int_{-1}^{1} \left[ \frac{ d ^{l+m}}{ dx^{l+m} }(x^{2}-1)^{l}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \right]dx \tag{4} \end{align*} $$
Let’s focus only on the integral part of the right-hand side. By resolving it with integration by parts, it becomes,
$$ \begin{align*} &\quad \ \int_{-1}^{1} \left[ \frac{ d ^{l+m}}{ dx^{l+m} }(x^{2}-1)^{l}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} \right]dx \\ &= \int_{-1}^{1} \left[ \frac{ d ^{l+m-1}}{ dx^{l+m-1} }(x^{2}-1)^{l}\right]^{\prime}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} dx \\ &= \left[ \frac{ d ^{l+m-1}}{ dx^{l+m-1} }(x^{2}-1)^{l}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l}\right]_{-1}^{1}-\int_{-1}^{1}\frac{ d ^{l+m-1}}{ dx^{l+m-1} }(x^{2}-1)^{l}\frac{ d ^{l-m+1}}{ dx^{l-m+1} }(x^{2}-1)^{l}dx \end{align*} $$
Here, the first term is $0$. $(x^{2}-1)^{l}$ is a polynomial of the degree $2l$and since $|m| \lt l$, both $l+m-1$ and $l-m$ are smaller than $l$, leaving at least $(x^{2}-1)$ not differentiated remaining. Inserting $\pm 1$ results in $0$. Continuing to solve the remaining terms by integration by parts,
$$ \begin{align*} &\quad \ -\int_{-1}^{1}\frac{ d ^{l+m-1}}{ dx^{l+m-1} }(x^{2}-1)^{l}\frac{ d ^{l-m+1}}{ dx^{l-m+1} }(x^{2}-1)^{l}dx \\ &= \left[- \frac{ d ^{l+m-2}}{ dx^{l+m-2} }(x^{2}-1)^{l}\frac{ d ^{l-m+1}}{ dx^{l-m+1} }(x^{2}-1)^{l} \right]+\int_{-1}^{1} \frac{ d ^{l+m-2}}{ dx^{l+m-2} }(x^{2}-1)^{l}\frac{ d ^{l-m+2}}{ dx^{l-m+2} }(x^{2}-1)^{l}dx \end{align*} $$
Here, the first term is $0$ for the reason mentioned above. By repeating this process of integration by parts $m$ times, the following equation is obtained.
$$ \int_{-1}^{1} \frac{ d ^{l+m}}{ dx^{l+m} }(x^{2}-1)^{l}\frac{ d ^{l-m}}{ dx^{l-m} }(x^{2}-1)^{l} dx=(-1)^{m}\int_{-1}^{1}\frac{ d ^{l}}{ dx^{l} }(x^{2}-1)^{l}\frac{d^{l}}{dx^{l}}(x^{2}-1)^{l}dx $$
Therefore, $(4)$ is as follows.
$$ \int_{-1}^{1}[P_{l}^{m}(x)]^{2}dx= \dfrac{1}{2^{2l}(l!)^{2}} \frac{(l+m)!}{(l-m)!}\int_{-1}^{1}\left[ \frac{ d ^{l}}{ dx^{l} }(x^{2}-1)^{l}\right]^{2}dx $$
$$ P_{l}(x)=\dfrac{1}{2^l l!} \dfrac{d^l}{dx^l}(x^2-1)^l $$
Then, by Rodrigues’ formula, it follows that.
$$ \begin{align*} \int_{-1}^{1}[P_{l}^{m}(x)]^{2}dx &= \dfrac{1}{2^{2l}(l!)^{2}} \frac{(l+m)!}{(l-m)!}2^{2l}(l!)^{2}\int_{-1}^{1}\left[ P_{l}(x)\right]^{2}dx \\ &= \frac{(l+m)!}{(l-m)!}\int_{-1}^{1}\left[ P_{l}(x)\right]^{2}dx \end{align*} $$
Finally, by the orthogonality of the Legendre polynomials, the following is concluded.
$$ \int_{-1}^{1}[P_{l}^{m}(x)]^{2}dx = \frac{2}{2l+1}\frac{(l+m)!}{(l-m)!} $$
■