Effect size and power | Statistics: concepts and methods

In short#

An effect estimate with a confidence interval shows the estimated magnitude and its uncertainty. A p-value alone does not.
For independent groups with a common variance, Cohen's d = (x̄₁ − x̄₂) / pooled SD. Standardization helps compare results when the effect measure and design are compatible, but hides the original units.
Power is the probability of rejecting H₀ for a specified effect under a model. It depends on the effect, variability, sample size, α and design. Plan sample size before collecting data.
For a non-significant result, a wide confidence interval shows uncertainty. The p-value does not prove there is no effect. Power is defined by a design and a specified alternative, not by the observed p-value.

In plain words#

A p-value describes how unusual the data would be under H₀ and the test assumptions. An effect estimate describes the size of the difference. A large sample can make a small difference statistically significant, but its practical importance depends on the field.

A confidence interval for a difference shows values compatible with the data under the chosen model. For the same two-sided test and its matching unadjusted 95% interval, excluding zero corresponds to p < 0.05.

Power is the probability that a test rejects H₀ when its model is correct and the effect equals a specified value. It applies to a particular test, effect and study design. Low power can explain a missed effect, but a non-significant p-value alone does not establish that.

In Linza#

For two independent groups, Linza shows the mean difference and 95% confidence interval. The standardized effect matches the test: Hedges g for Student and Bonett δ for Welch. For paired tests, it reports the mean paired difference and d_z.
Pairwise results include an estimate and interval when the selected method provides them. Regression reports a slope; 2 × 2 tables report an odds ratio.
One-way ANOVA reports η² and ω². Kruskal-Wallis reports ε²; its Dunn pair comparisons report rank-biserial r. Mann-Whitney also reports rank-biserial r, matched Wilcoxon reports matched-pairs rank-biserial r, and factorial effects report η²p. Plan sample size before collecting data.

Try it in Linza Opens in a new tab with synthetic data. Work in the original tab remains unchanged.

Sample size and power planning →

Common mistakes#

Reporting p without an estimate and interval.
Calculating power after analysis from the observed effect. This adds no information to the test. Report the effect estimate and confidence interval instead.

Check yourself#

Each of two groups has 5,000 participants. The mean difference is 0.2 points on a 0 to 100 scale, p < 0.001 and α = 0.05. Which conclusion follows?

Optional. Your answer is not saved.

The difference is necessarily important in practiceAt α = 0.05, the result is statistically significant, but the practical importance of 0.2 points needs separate assessmentThe probability of no difference is below 0.1%

Practice exercise: What is the upper bound of the 95% CI of the mean difference (standard minus new)? →

Sources#

Cohen J. (1988) Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum. Search for publication Opens in a new tab
Cumming G. (2014) The new statistics: why and how. Psychological Science, 25(1). Open publication Opens in a new tab
Hoenig J.M., Heisey D.M. (2001) The abuse of power: the pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1). Search for publication Opens in a new tab

References are for further reading. A method’s suitability depends on the study design and its assumptions.