l stoodialinza
← All topics

Step 2 · Drawing conclusions · Topic 4 of 17 · 8 min

Hypotheses, p-values and α

What statistically significant really means, and what a p-value does not tell you.

In short#

  • H₀ specifies the model being tested, often no effect, while H₁ describes an alternative. The test compares the data with what would be expected under H₀ and the stated assumptions.
  • Under H₀, p is the probability of obtaining a test statistic at least as extreme as the observed one. The selected test defines the direction and measure of extremeness.
  • Choose α before analysis. If H₀ is true and assumptions hold, the long-run false rejection rate does not exceed α. β is the probability of failing to reject H₀ under a specified alternative; power is 1 − β.
  • p is not the probability that H₀ is true, and it does not measure the size or practical importance of an effect. If p > α, the test did not meet the chosen threshold; this does not prove equality.

In plain words#

Suppose H₀ says that two population means are equal. If p = 0.03 and the assumptions hold, a result at least as extreme as the observed one would occur in about 3% of repetitions of this study. The chosen test determines which results count as extreme.

If α = 0.05 was chosen in advance, p = 0.01 meets that threshold, so the test rule rejects H₀. This does not show that the effect is large or important. With p = 0.3, the threshold is not met, but the absence of an effect has not been proved; the data may be insufficient.

Choose α, for example 0.05, before looking at the results. Selecting it afterward or trying analyses until p is small invalidates the claimed false positive rate.

In Linza#

  1. Set the significance level in Analyses before calculating. The default is α = 0.05. For multiple comparisons, check which p-values are adjusted by the selected correction.
  2. The conclusion is worded carefully: a difference is detected, or no difference is detected. p and significance stars appear next to it.
  3. When p is too small for machine arithmetic to represent, Linza reports this numerical limit and log(p) when available. This numerical limit does not mean absolute certainty.
Try it in Linza Opens in a new tab with synthetic data. Work in the original tab remains unchanged.

Common mistakes#

  • Trying tests and settings until p < 0.05 appears.
  • Writing that groups are equal when p = 0.3. At α = 0.05 the test did not meet its threshold, but equality requires a separate procedure, such as an equivalence test.
  • Believing that an effect with p = 0.001 matters more than one with p = 0.04.

Check yourself#

A test gives p = 0.03 with α = 0.05. Which conclusion is correct?

Optional. Your answer is not saved.

Practice exercise: Is a statistically significant difference detected at α = 0.05? →

Sources#

  • Wasserstein R.L., Lazar N.A. (2016) The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2). Open publication Opens in a new tab
  • Greenland S., Senn S.J., Rothman K.J. et al. (2016) Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology, 31. Search for publication Opens in a new tab

References are for further reading. A method’s suitability depends on the study design and its assumptions.