INTRODUCTION

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

STATISTICAL PROCEDURES are inherent to scientific discovery. As researchers, we use these procedures for two main reasons: to obtain point and interval estimates about the value of a population parameter, and to test the validity of a null hypothesis (5). Point and interval estimates emphasize the magnitude and uncertainty of the experimental results. The test of a null hypothesis helps guard against an unwarranted scientific conclusion, or it helps argue for a real experimental effect (18). When more than one hypothesis is testedwhen multiple comparisons are madethe validity of our scientific conclusions may be weakened if we fail to use an appropriate multiple comparison procedure (6, 8, 11, 14, 19, 20).

In studies published recently by the American Physiological Society (APS), the citation of a multiple comparison procedure is common (Table 1). This finding raises an important question: do physiologists understand the philosophies and assumptions behind competing multiple comparison procedures? This question is relevant for three reasons: there are many procedures available, textbooks of statistics (for example, Refs. 1, 13, and 18) provide little more than a cursory description of the procedures themselves, and there can be several solutions to the problem created by multiple comparisons.

In this paper, I summarize the statistical issue embedded in multiple comparisons, and I review the philosophies of handling this issue. Then, I illustrate the three proceduresNewman-Keuls, Bonferroni, least significant differencecited most often in my literature review. Last, I review the false discovery rate, a promising development in multiple comparisons.

Glossary

	Error rate for a single comparison
	Error rate for a family of k comparisons
H0	Null hypothesis
µ	Population mean
P	Achieved significance level
Pr{A}	Probability of event A
	Sample mean
*	Critical difference between two sample means

	THE ISSUE EMBEDDED IN MULTIPLE COMPARISONS

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

To test a null hypothesis, we must formulate the hypothesis beforehand. Then, using data collected during the experiment, we must compute the observed value T of some test statistic. Last, we must compare the observed value T to a critical value T*, chosen from the distribution of the test statistic that is based on the null hypothesis. If T is more extreme than T*, then that is surprising if the null hypothesis is true, and we are entitled to become skeptical about the scientific validity of the null hypothesis.

Suppose we want to assess renal blood flow in two independent samples. If our objective is to compare the underlying population means, µ₁ and µ₂, then one pair of null and alternative hypotheses, H₀ and H₁, is

The probability that we reject H₀ given that H₀ is true is the error rate . We can use mathematical notation¹ to write this statement as

(1)

Note that the critical value T* is the 100[1  ( / 2)]th percentile from the distribution of the test statistic given that the null hypothesis is true. Equation 1 can be rewritten as

(2)

Multiple comparisons. Suppose we want to assess renal blood flow in three independent samples.² In this setting, there are three alternative hypotheses, H₁-H₃, that correspond to the comparisons among population means:



Associated with each of these comparisons is an error rate of magnitude . If the three comparisons are considered to be a family, then the family will have an error rate , where  > . As a result, it is more likely that a true null hypothesis will be rejected erroneously. This is the statistical issue that lies at the heart of multiple comparison procedures.

Misguided multiple comparisons. In many of the studies tallied in Table 1, a multiple comparison procedure was used to analyze several groups of observations made on the same subjects. In general, this use of a multiple comparison procedure is misguided: most procedures assume that the groups are independent, but repeated observations on a subject, for example, observations made during baseline and then during several periods after some intervention, create correlation among the groups (9). As a result, the true error variability is underestimated, and the observed values for the standard deviations of the group means underestimate the true variabilities (9). When most multiple comparison procedures are used to analyze groups of repeated observations, the outcome will be an inflated number of statistically significant differences among the group means (see APPENDIX).

	PHILOSOPHIES ABOUT MULTIPLE COMPARISONS

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

Would you tell me, please, which way I ought to go from   here?Alice

  That depends a good deal on where you want to get   to.The Cat

L. Carroll in Alice's Adventures in Wonderland (1865)

When we decide the validity of a single comparison, we can make a mistake: we can reject a true null hypothesis, or we can fail to reject a false null hypothesis. When we decide the validity of k comparisonsthis happens in most experimentswe are more likely to reject a true null hypothesis. The challenge for any multiple comparison procedure is to satisfy two conflicting requirements: reduce the risk that we reject a true null hypothesis but maintain the likelihood that we detect an experimental effect if it exists (7, 12, 17). The relative importance assigned to these requirements has produced opposing philosophies about how to handle the issue of multiple comparisons.

Focus on individual comparisons. Proponents of this philosophy argue it is sufficient to control the single comparison error rate , the probability that we reject a true null hypothesis. They base this philosophy on the assumption that most scientific comparisons are preplanned (2, 15, 16). This assumption is naive and unrealistic: many experimental effects are discovered only after an investigator exploresrummages throughthe data.

Control for multiple comparisons. In general, physiologists examine the impact of an intervention on a seta familyof related comparisons: for example, the impact of some drug on renal blood flow and urinary excretion of hormones and electrolytes, or a series of paired comparisons among several groups of observations. In these situations, we base our scientific conclusions on a family of comparisons: that is, multiple comparisons considered as a single entity. As a result, it is not the single comparison error rate  that we must control but the family error rate , the probability that we reject at least one true null hypothesis in the family of comparisons (7, 8, 11-13, 17, 19-20). Multiple comparison procedures provide control of the family error rate .

	THE GENERAL STRATEGY

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

Most multiple comparison procedures use the same basic strategy: to make inferences about the population means for two groups, µ and µ, they compare the magnitude of the difference between the sample means and to a critical difference *. If
where

(3)

and where SE{u} is the standard error of the quantity u, then that is statistical evidence that µ  µ. Procedures differ in the statistics substituted for the coefficient c and the quantity u. Table 2 lists the statistics for the Newman-Keuls, Bonferroni, and least significant difference tests.

View this table: [in this window] [in a new window]   Table 2.   Calculation of the critical difference between sample means, *

	SIMULATED SAMPLE OBSERVATIONS

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

An article published recently in the Journal provides an ideal framework with which to illustrate multiple comparison procedures. In the experiment, Koch et al. (10) explored the heritability of running endurance, measured as distance run, in rats. I used the observed sample statistics from 10 experimental groups (Fig. 1) as the empirical foundation for the simulated sample observations.³

View larger version (11K): [in this window] [in a new window]   Fig. 1.   Experimental groups 1 - 10 associated with the simulated sample observations and derived sample statistics listed in Table 3. This diagram is based on the selective breeding procedure described in Ref. 10. The initial generation is generation 0. In each generation, the 2 female () and 2 male () rats at the extremes of observed running endurance were paired and bred to produce the subsequent generation.

This is how I generated the simulated sample observationsthe data. Let the random variable Y_j represent the distance run by a rat in group j, where j = 1, 2, ... , 10. Assume that each Y_j is distributed normally with mean µ_j and variance _j²
I estimated each µ_j and _j using approximate values for the observed group means and standard deviations (see Ref. 10, Tables 1 and 2). For simplicity, I limited each sample to 10 observations. One set of 10 simulated samples is listed in Table 3. For the rest of the review, I use the resulting sample means
and the resulting sample standard deviations
as the basis for my illustration of specific multiple comparison procedures.

	NEWMAN-KEULS PROCEDURE

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

The Newman-Keuls procedure⁴ is a multiple range test that compares the underlying population means of r experimental groups. That is, it evaluates the null hypothesis

(4)

The procedure sets the family error rate at , the single comparison error rate, by using studentized range distributions to calculate critical differences (see Eq. 5).

Another multiple range test is the Duncan procedure.⁵ It is only the specification of that differentiates the method of Duncan from that of Newman-Keuls. The Duncan family error rate is  = 1  (1  )^m1, where m is the number of means being compared. The Duncan multiple range test is a noted ancestor of modern multiple comparison procedures, but because grows with m, the test violates a basic tenet of multiple comparisons: the control of despite a large number of comparisons (see Ref. 12, p. 87-89).

The example. To make inferences about the equality of two population means, µ and µ, the Newman-Keuls procedure uses the critical difference ^*_m, defined as

(5)

In Eq. 5, the coefficient q_m, is the 100[1  ]th percentile from a studentized range distribution with m means and  degrees of freedom, and SE{} is the standard error of the sample mean. Using the pooled sample variance s² = 6,883 (see Table 3), the standard error of the sample mean is estimated as
Suppose we define  = 0.05. In this simulated experiment, there are  = 90 degrees of freedom (see Table 3). Because there can be groups of m = 2, 3, ... , 10 consecutive sample means, there are nine critical differences to be calculated using Eq. 5 (Table 4).

Practical considerations. The Newman-Keuls procedure evaluates all r (r  1)/2 paired comparisons among r sample means from a balanced design. The test assumes the r means are independent and are based on identical numbers of observations (Ref. 12, p. 86). When it compares more than three means, the Newman-Keuls procedure no longer caps the family error rate at ; instead,  >  (Ref. 8, p. 127). For this reason, the Newman-Keuls procedure is of limited value for multiple comparisons.

	BONFERRONI PROCEDURE

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

The Bonferroni inequality is a probability inequality that does control the family error rate . For a family of k comparisons, the Bonferroni inequality defines the upper bound of the family error rate to be
where  is the error rate for each comparison. In other words, the inequality assigns an error rate of / k to each comparison within the family. Because  can vary among comparisons, the general expression for the family error rate is

The example. To make inferences about the equality of two population means, µ and µ, the Bonferroni procedure relies on the critical difference *, defined as

(6)

In Eq. 6, the coefficient t _{/ 2,} is the 100[1  ( / 2)]th percentile from a t distribution with  degrees of freedom, and SE{  } is the standard error of the difference between the sample means.

/2,

(7)

Practical considerations. Although it is not a multiple comparison procedure per se, the Bonferroni inequality can be used for multiple comparison problems. The technique is valid regardless of whether the r sample means are independent or correlated (Ref. 12, p. 67). The Bonferroni inequality is appealing because it is versatile and simple. Unfortunately, its appeal is diminished by the strict protection of the single comparison error rate . As a consequence, the Bonferroni inequality is conservative: it will be unable to detect some of the actual differences among a family of k comparisons (see Table 5).

	LEAST SIGNIFICANT DIFFERENCE PROCEDURE

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

The least significant difference (LSD) procedure, developed by Sir R. A. Fisher, preceded the Newman-Keuls multiple range test. Like the Newman-Keuls test, the LSD procedure compares the underlying population means of r experimental groups (see Eq. 4), and it sets the family error rate at the single comparison error rate .

The example. To make inferences about the equality of two population means, µ and µ, the LSD procedure uses the critical difference *, defined as

(8)

In Eq. 8, the coefficient t _{/ 2,} is the 100[1  ( / 2)]th percentile from a t distribution with  degrees of freedom, and SE{  } is the standard error of the difference between the sample means.⁶

/ 2,

Practical considerations. The LSD procedure evaluates all r (r  1) /2 paired comparisons among r sample means. In its protected form, the procedure is done only if a preliminary analysis of variance is statistically significant (18). When it compares more than three means, the LSD procedure fails to maintain the family error rate at  (Ref. 8, p. 139). The solution to this problem is to replace t _{/ 2,} in Eq. 8 with a percentile from a studentized range distribution: q_r1, (Ref. 8, p. 139) or q_r, (Ref. 12, p. 92).⁷

	FALSE DISCOVERY RATE PROCEDURE: A RECENT DEVELOPMENT

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

In most experiments, scientists strive to make a discovery: to reject a null hypothesis. When an experiment involves a family of k comparisons, a scientist is more likely to make a mistaken discovery. The false discovery rate procedure⁸ is a promising solution to the problem of multiple comparisons. This procedure controls not the family error rate but the false discovery rate f, the expected fraction of null hypotheses rejected mistakenly
If all k null hypotheses are true,⁹ then f = ; if at least one null hypothesis is not true, then f   (3). When we define the family error rate , we also set an upper bound on the false discovery rate f. But if we control f rather than , we gain statistical power, the ability to detect an experimental effect if it exists (3, 4, 22).

The example. Unlike the preceding methods, the false discovery rate procedure operates on achieved significance levels (P values) to make inferences about a family of k comparisons. Let P_i represent the significance level associated with comparison i. To execute this procedure, we must complete three steps:

(9)

Practical considerations. Because the false discovery rate procedure operates on actual P values, it is quite versatile. For example, the procedure can be employed when a family of k comparisons involves different test statistics such as Student t and Wilcoxon signed rank statistics (3, 4). The false discovery rate procedure is valid when the k comparisons are independent (a sample mean is part of only one comparison) or correlated (a sample mean is part of more than one comparison, as in the example) (3, 4, 22).

	SUMMARY

TOP ABSTRACT INTRODUCTION THE ISSUE EMBEDDED IN... PHILOSOPHIES ABOUT MULTIPLE... THE GENERAL STRATEGY SIMULATED SAMPLE OBSERVATIONS NEWMAN-KEULS PROCEDURE BONFERRONI PROCEDURE LEAST SIGNIFICANT DIFFERENCE... FALSE DISCOVERY RATE PROCEDURE:... SUMMARY APPENDIX REFERENCES

We dare not seek a single multiple comparison procedure for all experiments.

Adapted from John W. Tukey (1994)

This remark, written by a pioneer in the area of multiple comparisons, reflects the range of multiple comparison problems that manifest themselves in scientific research. Over the last 50-60 years, statisticians have explored numerous approaches in an effort to address these problems (8, 12). In physiology, as in other disciplines, experiments that involve problems of multiple comparisons are common.

In this review, I have shown that, as researchers, we are more likely to reject a true null hypothesis if we fail to use a multiple comparison procedure when we analyze a family of comparisons. I have also illustrated the three procedures cited most often in APS journals: Newman-Keuls, Bonferroni, and LSD. Unfortunately, each of these is of limited value. In many experimental situations, the Newman-Keuls and LSD procedures fail to control the family error rate, the probability that we reject at least one true null hypothesis. In contrast, the Bonferroni inequality is overly conservative: it fails to detect some of the actual differences that exist within the family.

Finally, I have reviewed the false discovery rate: a versatile, simple, and powerful approach to multiple comparisons. As Tukey suggests, it is perhaps unrealistic to expect that a single multiple comparison procedure will suffice for all situations: a statistical procedure designed specifically for a particular experimental situation will perform better than a general procedure. Nevertheless, there is growing evidence (4, 22) that the false discovery rate procedure may be the best practical solution to the problems of multiple comparisons that exist within science.