Hypothesis Testing

Concepts

Null Hypothesis $H_0$ and Alternative Hypothesis $H_1$

• Null hypothesis $H_0$: Initial claim / assumption on the values that $\mu$ can take
  • 通俗来说：没变化 / 维持现状
• Alternative hypothesis $H_1$: The competing claim / assumption against which we would like to test the initial claim / assumption
  • 通俗来说：想证明的新发现

• Simple hypothesis: Hypothesis in which $\mu$ only takes on 1 value
• Composite hypothesis: Hypothesis in which $\mu$ takes on a range of value

Example

Let $X$ equal the breaking strength of a steel bar.
If the bar is manufactured by process I, $X\sim N(50,36)$.
It is hoped that if process II (a new process) is used, $X\sim N(55,36)$.
Given a large number of steel bars manufactured by process II, how could we test whether the five-unit increase in the mean breaking strength was realized?

Judgement: 5-unit increase in $\mu$ or not

• Simple null hypothesis $H_0$: $\mu =50$
• Simple alternative hypothesis $H_1$: $\mu =55$
• Composite null hypothesis $H_0$: $\mu \le 50$
• Composite alternative hypothesis $H_1$: $\mu >50$

Critical Region $C$ and Test Statistic $T$

The space of the sample $\mathcal{D}:=\{(x_1,x_2,\dots ,x_n)\mid x_i\in S_X,i=1,2,\cdots ,n\}$

Hypothesis testing is based on critical region $C$, which is subset of $\mathcal{D}$.

• Reject $H_0$ (accept $H_1$) if $(x_1,x_2,\cdots ,x_n)\in C$
• Fail to reject $H_0$ (reject $H_1$) if $(x_1,x_2,\cdots ,x_n)\notin C$

Test statistic $T$ is quantity derived from the sample for hypothesis testing.

The critical region $C$ is usually specified in terms of test statistic $T$.

In the #Example,

• Test statistic $T=\overline{X}$
• Critical region $C=\{(x_1,x_2,\cdots ,x_n)\in \mathcal{D}\mid \overline{x}\ge 53\}$

Errors in Hypothesis Testing

• Type I error: False positive 误报

• Type II error: False negative 漏报

Example

$X_1,\dots ,X_n\stackrel{i.i.d.}{\sim }N(\mu ,36)$，$n=16$
Take critical region $C=\{\overline{x}\ge 53\}$.
What is the probability of type I error and type II error in the following test

$$H_0:\mu =50\text{against}{H}_1:\mu =55$$

Significance Level

We usually fix the probability of type I error $\alpha$, and then determine the corresponding critical region.
In this scenario, the value of $\alpha$ is called significance level of the test.

Example

$X_1,\dots ,X_n\stackrel{i.i.d.}{\sim }N(\mu ,36)$，$n=16$
Take $\alpha =0.05$
Find the critical region

$p$-value: Observed Significance Level

$p$-value: If $H_0$ is true, the probability of observing a more extreme test statistic (favour $H_1$)
Smaller $p$-value indicates the data is more unlikely coming from $H_0$ and $H_1$ is perhaps better retained

$H_0$ is rejected if $p$-value $\le \alpha$

Example

Following the same example above (${\sigma }^2=36$，$n=16,\alpha =0.05$).
Given observed test statistics $\overline{x}=56$.

• For $H_0:\mu =50$ versus $H_1:\mu =55$
  • 更极端方向是 $\overline{X}\ge 56$

• For $H_0:\mu =55$ versus $H_1:\mu >55$
  • 更极端方向是 $\overline{X}\ge 56$

• For $H_0:\mu =55$ versus $H_1:\mu <55$
  • 更极端方向是 $\overline{X}\le 56$

• For $H_0:\mu =55$ versus $H_1:\mu \ne 55$
  • 更极端方向是 $\overline{X}\ge 56$ and $\overline{X}\le 54$

Hypothesis Testing for Normal Mean

There are 3 approaches to hypothesis testing

• $p$-value approach
  • Reject $H_0$ if $p$-value $\le \alpha$
• Critical region approach
  • Reject $H_0$ if statistic falls in critical region
• Confidence interval approach
  • Reject $H_0$ if null value (value of null hypothesis) is not include in CI

Case 1: ${\sigma }^2$ Known

Consider test statistic

Under $H_0:\mu ={\mu }_0$, $T\sim N(0,1)$

Once the data $x_1,\cdots ,x_n$ is observed, we can compute

For the one-sided test $H_0:\mu ={\mu }_0,H_1:\mu \boxed{{\displaystyle >}}{\mu }_0$

• $p$-value is

• Reject $H_0$ at significance level $\alpha$ when either
  • $p$-value approach: $p\le \alpha$ OR
  • Critical region approach: $z\ge z_{\alpha }$ OR
  • Confidence interval approach: ${\mu }_0\notin (\overline{x}-z_{\alpha }\frac{\sigma }{\sqrt{n}},\mathrm{\infty })$

For one-sided test $H_0:\mu ={\mu }_0,H_1:\mu \boxed{{\displaystyle <}}{\mu }_0$

Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $z\le -z_{\alpha }$ OR
• ${\mu }_0\notin (-\mathrm{\infty },\overline{x}+z_{\alpha }\frac{\sigma }{\sqrt{n}})$

For two-sided test $H_0:\mu ={\mu }_0,H_1:\mu \ne {\mu }_0$

Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $|z|\ge z_{\frac{\alpha }{2}}$ OR
• ${\mu }_0\notin (\overline{x}-z_{\frac{\alpha }{2}}\frac{\sigma }{\sqrt{n}},\overline{x}+z_{\frac{\alpha }{2}}\frac{\sigma }{\sqrt{n}})$
  

Case 2: ${\sigma }^2$ Unknown

When ${\sigma }^2$ is unknown, replace standard normal distribution by $t$-distribution with degree of freedom $(n-1)$ and ${\sigma }^2$ by $S^2$ in #Case 1 $ sigma 2$ Known

Consider test statistic

Under $H_0:\mu ={\mu }_0$, $T\sim t(n-1)$
Once data $x_1,\cdots ,x_n$ are observed, we can compute

For the one-sided test $H_0:\mu ={\mu }_0,H_1:\mu \boxed{{\displaystyle >}}{\mu }_0$

Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $t\ge t_{\alpha }(n-1)$ OR
• ${\mu }_0\notin (\overline{x}-t_{\alpha }(n-1)\frac{s}{\sqrt{n}},\mathrm{\infty })$

For the one-sided test $H_0:\mu ={\mu }_0,H_1:\mu \boxed{{\displaystyle <}}{\mu }_0$

Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $t\le -t_{\alpha }(n-1)$
• ${\mu }_0\notin (-\mathrm{\infty },\overline{x}+t_{\alpha }(n-1)\frac{s}{\sqrt{n}})$

For the two-sided test $H_0:\mu ={\mu }_0,H_1:\mu \ne {\mu }_0$

Reject $H_0$ at significance level $\alpha$ when either

• $2Pr(T>|t|;T\sim t(n-1))\le \alpha$
• $|t|\ge t_{\frac{\alpha }{2}}(n-1)$
• ${\mu }_0\notin (\overline{x}-t_{\frac{\alpha }{2}}(n-1)\frac{s}{\sqrt{n}},\overline{x}+t_{\frac{\alpha }{2}}(n-1)\frac{s}{\sqrt{n}})$

Example

Two-sided Test

Let $X$ (in millimeters) equal the growth in 15 days of a tumor induced in a mouse.
Assume that the distribution of $X$ is $N(\mu ,{\sigma }^2)$
Test the null hypothesis $H_0:\mu =4.0$ against the two-sided alternative hypothesis $H_1:\mu \ne 4.0$
Observe that $n=9$ observations with $\overline{x}=4.3,s=1.2$ and set significance level of $\alpha =0.10$

Hence, we fail to reject $H_0$ at 10% significance level

One-sided Test

Suppose that we have $n=25,\overline{x}=308.8,s=115.15$
Test $H_0:\mu =500$ against $H_1:\mu <500$, given significance level $\alpha =0.01$

Test Equality of Two Normal Means

Settings

• 2 populations with i.i.d. samples
  • $X_1,\cdots ,X_n\sim N({\mu }_X,{\sigma }_X^2)$
  • $Y_1,\cdots ,Y_m\sim N({\mu }_Y,{\sigma }_Y^2)$
• Test $H_0:{\mu }_X={\mu }_Y$ versus $H_1$
  3 cases:
• Pooled $t$-test
  • $X_1,\cdots ,X_n$ and $Y_1,\cdots ,Y_m$ are independent
  • ${\sigma }_X^2={\sigma }_Y^2={\sigma }^2$ are unknown
• Welch's $t$-test
  • $X_1,\cdots ,X_n$ and $Y_1,\cdots ,Y_m$ are independent
  • ${\sigma }_X^2\ne {\sigma }_Y^2$ are unknown
• Paired $t$-test
  • $n=m$ and $(X_i,Y_i)$ are independent pairs

Pooled $t$-test

Consider the pooled $t$-test statistic

The pooled variance $S_p^2$ is

Under $H_0:{\mu }_X={\mu }_Y$, $T\sim t(n+m-2)$

Once data $x_1,\cdots ,x_n$ and $y_1,\cdots ,y_m$ is observed, we can compute

For one-sided test $H_0:{\mu }_X={\mu }_Y$ versus $H_1:{\mu }_X\boxed{{\displaystyle >}}{\mu }_Y$
$p$-value is

Decision rules: Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $t\ge t_{\alpha }(n+m-2)$ OR
• $0\notin (\overline{x}-\overline{y}-t_{\alpha }(n+m-2)s_p\sqrt{\frac{1}{n}+\frac{1}{m}},\mathrm{\infty })$

For one-sided test $H_0:{\mu }_X={\mu }_Y$ versus $H_1:{\mu }_X\boxed{{\displaystyle <}}{\mu }_Y$
$p$-value is

Decision rules: Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $t\le -t_{\alpha }(n+m-2)$ OR
• $0\notin (-\mathrm{\infty },\overline{x}-\overline{y}+t_{\alpha }(n+m-2)s_p\sqrt{\frac{1}{n}+\frac{1}{m}})$

For two-sided test $H_0:{\mu }_X={\mu }_Y$ versus $H_1:{\mu }_X\ne {\mu }_Y$
$p$-value is

Decision rules: Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$
• $|t|\ge t_{\frac{\alpha }{2}}(n+m-2)$
• $0\notin (\overline{x}-\overline{y}-t_{\frac{\alpha }{2}}{s}_p\sqrt{\frac{1}{n}+\frac{1}{m}},\overline{x}-\overline{y}+t_{\frac{\alpha }{2}}{s}_p\sqrt{\frac{1}{n}+\frac{1}{m}})$

Example

Suppose $X\sim N({\mu }_X,{\sigma }^2)$ and $Y\sim N({\mu }_Y,{\sigma }^2)$.
Observe that

$$\begin{array}{rlrl}n& =12,& \overline{x}=1076.75,& s_X^2=29.30\\ m& =12,& \overline{y}=1072.33,& s_Y^2=26.24\end{array}$$

We would like to test $H_0:{\mu }_X={\mu }_Y$ versus $H_1:{\mu }_X\ne {\mu }_Y$ at significance level $\alpha =0.10$

Compute test statistic

Hence, reject $H_0$ at $10\mathrm{\%}$ significant level.

Welch's $t$-test

Consider the Welch's $t$-test statistic

Under $H_0$, $T\sim t(r)$ approximately with

Once data $x_1,\cdots ,x_n$ and $y_1,\cdots ,y_m$, we can compute

For two-sided test $H_0:{\mu }_X={\mu }_Y$ versus $H_1:{\mu }_X\ne {\mu }_Y$
$p$-value is

Decision rules: Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $|t|\ge t_{\frac{\alpha }{2}}(r)$ OR
• $0\notin (\overline{x}-\overline{y}-t_{\frac{\alpha }{2}}(r)\sqrt{\frac{s_X^2}{n}+\frac{s_Y^2}{m}},\overline{x}-\overline{y}+t_{\frac{\alpha }{2}}(r)\sqrt{\frac{s_X^2}{n}+\frac{s_Y^2}{m}})$

Paired $t$-test

Consider the following two-sided test

Let $D_i:=X_i-Y_i$, for $i=1,2,\cdots ,n$
Consider the paired statistic

Under $H_0:{\mu }_X={\mu }_Y$, $T\sim t(n-1)$

Once data $x_1,\cdots ,x_n$ and $y_1,\cdots ,y_n$ are observed, we can compute

$p$-value is

Decision rules: Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $|t|\ge t_{\frac{\alpha }{2}}(n-1)$ OR
• $0\notin (\overline{x}-\overline{y}-t_{\frac{\alpha }{2}}(n-1)\frac{s_D}{\sqrt{n}},\overline{x}-\overline{y}+t_{\frac{\alpha }{2}}(n-1)\frac{s_D}{\sqrt{n}})$

Example

24 girls in the 9th and 10th grades were put on an ultraheavy rope-jumping program.
Someone thought that such a program would increase their speed in the 40-yard dash.
Let $D$ equal the difference in time to run the 40-yard dash-the "before-program time" ($X$) minus the "after-program time" ($Y$).
Assume that the distribution of $D$ is approximately $N({\mu }_d,{\sigma }_d^2)$.
We shall test $H_0:{\mu }_D=0$ against $H_1:{\mu }_D>0$ at significance level $\alpha =0.10$.
We observe $n=24$, $\overline{d}=0.0788$, $s_D=0.2549$.

Test Equality of Two Normal Variances

Suppose two independent populations with i.i.d. samples

• $X_1,\cdots ,X_n\sim N({\mu }_X,{\sigma }_X^2)$
• $Y_1,\cdots ,Y_m\sim N({\mu }_Y,{\sigma }_Y^2)$

To test $H_0:{\sigma }_X^2={\sigma }_Y^2$ versus $H_1:{\sigma }_X^2\ne {\sigma }_Y^2$ ,
we have

Recall $F$-distribution definition,

Define test statistic

Under $H_0:{\sigma }_X^2={\sigma }_Y^2$, the ratio becomes

Once data $x_1,\cdots ,x_n$ and $y_1,\cdots ,y_m$ is observed, we can compute

Reject $H_0$ at significance level $\alpha$ when

For two-sided test $H_0:{\sigma }_X^2={\sigma }_Y^2$ versus $H_1:{\sigma }_X^2\ne {\sigma }_Y^2$

• When $f>1$, $p$-value is

• When $f<1$, $p$-value is

Decision rules: Reject $H_0$ at significance level $\alpha$ when either

• $p\le \alpha$ OR
• $f\ge F_{\frac{\alpha }{2}}(n-1,m-1)\text{ or }f\le F_{1-\frac{\alpha }{2}}(n-1,m-1)$
• $1\notin \underset{\text{two-sided }100(1-\alpha )\mathrm{\%}\text{ CI of }\frac{{\sigma }_Y^2}{{\sigma }_X^2}}{\underbrace{[\frac{s_Y^2}{s_X^2}{F}_{1-\frac{\alpha }{2}}(n-1,m-1),\frac{s_Y^2}{s_X^2}{F}_{\frac{\alpha }{2}}(n-1,m-1)]}}$

Hypothesis Testing for Proportions

Let $p$ be the proportion of population who has certain characteristic
To estimate $p$, draw $n$ i.i.d. random samples

where each $X_i$ is $1$ when it has the characteristic and $0$ if doesn't have

Denote $Y=\sum _{i=1}^n{X}_i\sim \text{Bin}(n,p)$, then

If $n$ is large enough, by CLT,

Given null hypothesis test $H_0:p=p_0$, construct test statistic as follow

Under $H_0$, $T\sim N(0,1)$ approximately

Once data $x_1,\cdots ,x_n$, are observed, we can compute

$t$ is the realized value of $T$ (即 $t$ 是 $T$ 的观测值)

For one-sided test: $H_0:p=p_0,H_1:p\boxed{{\displaystyle >}}{p}_0$
$p$-value is

where $Z\sim N(0,1)$

Reject $H_0$ at significance level $\alpha$ when either

• $p$-value $p\le \alpha$ OR
• $t\ge z_{\alpha }$ OR
• $\hat{p}\ge p_0+z_{\alpha }\sqrt{\frac{p_0(1-p_0)}{n}}$

For one-sided test: $H_0:p=p_0,H_1:p\boxed{{\displaystyle <}}{p}_0$
$p$-value is

Reject $H_0$ at significance level $\alpha$ when either

• $p$-value $p\le \alpha$ OR
• $t\le -z_{\alpha }$ OR
• $\hat{p}\le p_0-z_{\alpha }\sqrt{\frac{p_0(1-p_0)}{n}}$

For two-sided test $H_0:p=p_0,H_1:p\ne p_0$
$p$-value is

Reject $H_0$ at significance level $\alpha$ when either

• $p$-value $p\le \alpha$ OR
• $|t|\ge z_{\frac{\alpha }{2}}$ OR
• $\hat{p}\ge p_0+z_{\frac{\alpha }{2}}\sqrt{\frac{p_0(1-p_0)}{n}}$ or $\hat{p}\le p_0-z_{\frac{\alpha }{2}}\sqrt{\frac{p_0(1-p_0)}{n}}$

Example

It was claimed that many commercially manufactured dice are not fair because the “spots” are really indentations, so that, for example, the 6-side is lighter than the 1-side.
Let $p$ equal the probability of rolling a 6 with one of these dice.
To test $H_0:p=\frac{1}{6}$ against the alternative hypothesis $H_1:p\ne \frac{1}{6}$ with significance level $\alpha =5\mathrm{\%}$, several such dice will be rolled to yield a total of $n=8000$ observations, and the number 6 appears 1389 times.

Let $Y$ be the no. of times that 6 resulted in the 8000 trials.
Then, the test statistics is

• Critical region approach: $|t|=1.67<1.96$
  • Hence, fail to reject $H_0$ at $5\mathrm{\%}$ significance level
• $p$-value approach

- Hence, fail to reject $H_0$ at $5\mathrm{\%}$ significance level
If set $\alpha =0.1$ with $z_{0.05}=1.64$, $H_0$ will be rejected at $10\mathrm{\%}$ significance level

Hypothesis Testing for Equality of Two Proportions

Given there are 2 populations, namely population 1 and population 2.
Let $p_1$ and $p_2$ be the proportion of population 1 and 2 respectively, who has a certain characteristic.

Test on $H_0:p_1=p_2$ against possible alternative hypothesis $H_1$

Denote the samples to be

Assume they're independent

Denote the sums to be

MLE of $p_1,p_2$ are

If $n,m$ are large enough, then by CLT,

For two-sided test $H_0:p_1=p_2,H_1:p_1\ne p_2$
Under $H_0$, $p_1=p_2$ and