l stoodialinza
← All topics

Step 2 · Drawing conclusions · Topic 7 of 17 · 7 min

Multiple comparisons

Many tests increase the chance of at least one false finding. Choose a correction for a family of hypotheses defined in advance.

In short#

  • If all m independent hypotheses are true and each test uses level α, the chance of at least one false rejection is 1 − (1 − α)ᵐ. For m = 10 and α = 0.05, it is about 0.40.
  • Bonferroni controls the family-wise error rate (FWER), the probability of at least one Type I error in a hypothesis family, under any dependence using a threshold of α/m. For independent tests, the exact Sidak threshold is 1 − (1 − α)^(1/m). Tukey compares all pairs; Dunnett compares each group with a control under a common-variance model.
  • Define the family before analysis: which conclusions should share error control. Its composition and test dependence affect the choice of correction.
  • An omnibus ANOVA, Kruskal-Wallis or log-rank test evaluates differences across groups overall, but does not show which pairs differ.
  • Holm sorts raw p-values in ascending order. For the value at position i, it computes p × (m − i + 1), takes the cumulative maximum, caps the result at 1, and maps it back to the original hypotheses.

In plain words#

If all 20 null hypotheses are true and each test uses α = 0.05, there will be one false rejection on average. The chance of at least one finding also depends on how tests are related; the independent-test formula is not universal.

Corrections control the chance of at least one false rejection in a family, but differ in assumptions and sensitivity. Bonferroni and Holm work under arbitrary dependence. Tukey is designed for all pairwise comparisons; Dunnett is for comparisons with one control under a common-variance model.

Holm tests hypotheses step by step: compare the smallest p with α/m, then the next with α/(m − 1). Stop rejecting hypotheses after the first non-rejection. Adjusted p-values express this same rule for each hypothesis.

In Linza#

  1. For ordinary one-way ANOVA, choose Tukey, Dunnett, Bonferroni or Sidak. Welch ANOVA supports Bonferroni and Sidak. The comparison table shows the selected correction.
  2. After Kruskal-Wallis, Linza runs pairwise Dunn tests. Choose Bonferroni, Sidak or Holm for the family of pairs.
  3. The report names the selected correction and comparison family. Describe the planned comparisons and multiplicity procedure in the Methods section.
  4. For Dunn p-values with Holm correction, sort values in ascending order and multiply by m, m − 1, down to 1. Then take the cumulative maximum, cap values at 1 and map them back to the original pairs.
Try it in Linza Opens in a new tab with synthetic data. Work in the original tab remains unchanged.

Common mistakes#

  • Running t-tests between all pairs without a correction.
  • Choosing the correction after seeing the results.

Check yourself#

In an ANOVA with a common variance, five treatments are compared with one control as planned. Which procedure is designed for this comparison family?

Optional. Your answer is not saved.

Sources#

  • Tukey J.W. (1949) Comparing individual means in the analysis of variance. Biometrics, 5(2). Open publication Opens in a new tab
  • Dunnett C.W. (1955) A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association, 50(272). Search for publication Opens in a new tab
  • Bender R., Lange S. (2001) Adjusting for multiple testing: when and how? Journal of Clinical Epidemiology, 54(4). Search for publication Opens in a new tab
  • Holm S. (1979) A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2). Search for publication Opens in a new tab

References are for further reading. A method’s suitability depends on the study design and its assumptions.