8.2 Type I and Type II Errors

In testing hypotheses, there are only two possible outcomes: either reject $H_0$ or do not reject $H_0$; in reality, there are only two possible scenarios: either $H_0$ is true or $H_0$ is false. Hence, regardless of which conclusion we make, we have a chance to make an error. There are two types of errors: Type I and Type II.

Type I error: reject the null $H_0$ when $H_0$ is in fact true.

Type II error: do not reject the null $H_0$ when $H_0$ is false.

Table 8.2: Type I and Type II Error

The probability of type I error is denoted as $\alpha$, and the probability of type II error is denoted as $\beta$. That is:

$\alpha =P(\text{Type I error})=P(\text{Reject }{H}_0|H_0\text{ is true})$

$\beta =P(\text{Type II error})=P(\text{Do not reject }{H}_0|H_0\text{ is false})$

The type I error rate $\alpha$ is also called the significance level of a hypothesis test.

Example: Type I and Type II Errors

In a diabetes blood test, a patient is diagnosed with the disease if the sugar level in their bloodstream is larger than the threshold C=130 mg/dL. Suppose the distributions of sugar levels for the two populations (diabetes-free and having diabetes) are the two bell-shaped curves shown in the following figure.

Define the hypotheses: $H_0$ (a patient is disease free) vs. $H_a$ (a patient has diabetes). What are the type I and type II errors in this example?

Type I error: claim the person has diabetes (reject the null $H_0$) but actually the person does not have diabetes ($H_0$ is in fact true). This is often referred to as a false positive.Type II error: claim the person does not have diabetes(do not reject the null $H_0$), but actually the person has diabetes ($H_0$ is false). This is often referred to as a false negative.

The figure in the above example shows the trade-off between type I and type II errors. The gold area gives $\alpha$, the probability of the type I error; and the blue area gives $\beta$, the probability of the type II error. If we increase the threshold C (move the cut-off to the right), the gold area will reduce and the blue area will increase. That is the type I error rate $\alpha$ will decrease and the type II error rate $\beta$ will increase. On the other hand, if we reduce the threshold C (move the cut-off to the left), the type I error rate $\alpha$ will increase and the type II error rate will decrease. This is the trade-off between the type I and type II errors $\alpha$ and $\beta$. It is not a good idea to set either $\alpha$ or $\beta$ to be too close to 0; otherwise, the other error rate will be huge. For example, if we set the threshold C very large, few individuals will be diagnosed as diabetic; as a result, many diabetic individuals will be misclassified as not having the disease (meaning we have a high probability of committing a type I error). On the other hand, if we set the threshold C very small, most individuals will be diagnosed as diabetic; consequently, many individuals who are free of diabetes will be misclassified as diabetic (meaning we have a high probability of committing a type II error). In general, we can set $\alpha$ (or $\beta$) to be relatively small if the consequence of the type I (or type II) error is more serious. The power of a test is defined as

$1-\beta =1-P(\text{Type II error})$ $=1-P(\text{Do not reject }{H}_0|H_0\text{ false})=P(\text{Reject }{H}_0|H_0\text{ false}).$

This is the probability that we reject $H_0$ when $H_0$ is false. Thus, it is of interest for a statistical test to have a high level of power.