Test of Independence 📂Statistical Testing

Test of Independence

Hypothesis Test ¹

In a multinomial experiment, suppose the data have two characteristics $X$ and $Y$, where $X$ has $R$ categories and $Y$ has $C$ categories, and that this categorical data is obtained through $n$ independent trials. The following hypothesis test using the Pearson chi-squared test statistic is called the test of independence.

$H_{0}$: The two categories are independent.
$H_{1}$: The two categories are dependent.

Test Statistic

The test statistic uses the Pearson chi-squared test statistic. $$ \mathcal{X}^{2} = \sum_{i=1}^{R} \sum_{j=1}^{C} {{ \left( O_{ij} - E_{ij} \right)^{2} } \over { E_{ij} }} $$ Here, the frequency $O_{ij}$ of data for which $X$ belongs to the $i$-th category and $Y$ belongs to the $j$-th category is called the observed frequency, and the ratio $E := r_{i} c_{j} / n$ of the sample size $n$ to the product $r_{i} c_{j}$—where $r_{i}$ is the number of samples for which $X$ belongs to the $i$-th category and $c_{j}$ is the number of samples for which $Y$ belongs to the $j$-th category—is called the expected frequency. This test statistic follows a chi-squared distribution, whose degrees of freedom is $(R-1)(C-1)$, the product of the values obtained by subtracting $1$ from each of the numbers of categories $R$ and $C$ for $X$ and $Y$.

Explanation

Degrees of Freedom

Simply let $p_{ij}$ denote the ratio obtained by dividing by $n$ the frequency $O_{ij}$ of data for which $X$ belongs to the $i$-th category and $Y$ belongs to the $j$-th category. $$ \begin{align*} p_{i} = \sum_{j1=1}^{C} p_{ij} = {\frac{ r_{i} }{ n }} \implies & \sum_{i=1}^{R} p_{i} = 1 \\ p_{j} = \sum_{i=1}^{R} p_{ij} = {\frac{ c_{j} }{ n }} \implies & \sum_{j=1}^{C} p_{j} = 1 \end{align*} $$ Considering each marginal probability, the constraints above follow, so if we know $R-1$ and $C-1$ probabilities in each category, the remaining one is effectively determined. Therefore the degrees of freedom must be their product $(R - 1) (C - 1)$.

X\Y	y1	y2
x1	0000	0000
x2	0000	0000
x3	0000	0000

For example, the contingency table above represents data in which $X$ has $R = 3$ categories and $Y$ has $C = 2$ categories, and its degrees of freedom is $(3-1)(2-1) = 2$.

Independence

It may look difficult because of all the talk of contingency tables and products of probabilities, but essentially the test of independence is an extension of the goodness-of-fit test. This is because the null hypothesis of the goodness-of-fit test is that ’the given data was sampled so as to conform to the theoretical probabilities,’ and in the test of independence that theoretical probability simply appears as the product of the marginal probabilities for each category.

Independence of events: Suppose a probability space $(\Omega , \mathcal{F} , P)$ is given.
For $P(B)>0$, $\displaystyle P (A | B) = {{P(A \cap B)} \over {P(B)}}$ is called the conditional probability of $A$ given $B$.
If $P(A | B) = P(A)$, i.e. $P( A \cap B) = P(A) \cdot P(B)$, then $A, B$ are said to be independent of each other.

In fact, the independence of events is defined by the fact that the probability of two events occurring simultaneously equals the product of the probabilities of each occurring. In the context of the test of independence, if the null hypothesis is true, then $p_{ij} = p_{i} p_{j}$, so the following relation must hold. $$ \begin{align*} & {\frac{ O_{ij} }{ n }} = p_{ij} = p_{i} p_{j} \\ \implies & O_{ij} = n p_{i} p_{j} = n {\frac{ r_{i} }{ n }} {\frac{ c_{j} }{ n }} = {\frac{ r_{i} c_{j} }{ n }} = E_{ij} \end{align*} $$ A large difference between the observed frequency $O_{ij}$ and the expected frequency $E_{ij}$ means that the value of the chi-squared test statistic becomes large—in other words, the value of $\mathcal{X}^{2}$, which should have been small if the two characteristics were truly independent in theory, is large—so we must doubt the null hypothesis itself, namely $p_{ij} = p_{i} p_{j}$.

Example

Mendel’s Laws of Inheritance

(Continuing from the goodness-of-fit test of populations post)

According to Mendel’s laws of inheritance, as shown above, when yellow is dominant and green is recessive with respect to color, and round is dominant and wrinkled is recessive with respect to shape, if a purebred that is yellow and round is crossed with a purebred that is green and wrinkled, the phenotype of the first generation follows the dominant traits and is yellow and round, while in the second generation yellow and green appear in a 3:1 ratio and round and wrinkled appear in a 3:1 ratio, so that overall a 9:3:3:1 ratio appears².

	Yellow-Round	Yellow-Wrinkled	Green-Round	Green-Wrinkled
Observed frequency	773	231	238	59

If the observed frequencies of the second-generation peas thus obtained are as above ³, then the total number of samples is $n = 1301$ and the number of categories is $k = 4$, but in the test of independence we must construct a contingency table as follows.

Color\Shape	Round	Wrinkled
Yellow	773	231
Green	238	59

Now let us check, at significance level $\alpha = 0.05$, whether the two characteristics color and shape are independent of each other. Note that in the hypothesis test, the degrees of freedom of the chi-squared distribution is not $k-1 = 3$; since $R = 2$ and $C = 2$, it is $(2-1)(2-1) = 1$.

$H_{0}$: In the experimental results, color and shape are independent of each other.
$H_{1}$: In the experimental results, color and shape are dependent on each other.

First, the null hypothesis and alternative hypothesis are as above, and its Pearson chi-squared test statistic is computed as follows. $$ \begin{align*} E_{11} =& {\frac{ (773 + 231) \cdot (773 + 238) }{ 1301 }} \approx 780.2029 \\ E_{12} =& {\frac{ (773 + 231) \cdot (231 + 59) }{ 1301 }} \approx 223.7971 \\ E_{21} =& {\frac{ (238 + 59) \cdot (773 + 238) }{ 1301 }} \approx 230.7971 \\ E_{22} =& {\frac{ (238 + 59) \cdot (231 + 59) }{ 1301 }} \approx 66.2029 \\ \mathcal{X}^{2} =& \sum_{i=1}^{R} \sum_{j=1}^{C} {{ \left( O_{ij} - E_{ij} \right)^{2} } \over { E_{ij} }} \\ \approx & {\frac{ \left( 773 - 780.2029 \right)^{2} }{ 780.2029 }} + {\frac{ \left( 231 - 223.7971 \right)^{2} }{ 223.7971 }} \\ & + {\frac{ \left( 238 - 230.7971 \right)^{2} }{ 230.7971 }} + {\frac{ \left( 59 - 66.2029 \right)^{2} }{ 66.2029 }} \\ \approx & 1.31 \end{align*} $$ For the given significance level $\alpha = 0.05$, the upper bound of the rejection region following the chi-squared distribution with $1$ degree of freedom is $\chi_{1 - \alpha}^{2} (1) \approx 3.84$, and since $\chi_{1 - \alpha}^{2} > \mathcal{X}^{2}$, we cannot reject the null hypothesis. According to the goodness-of-fit test we could not support Mendel’s laws of inheritance through this data, but we can now claim that the two traits are independent.