Tukey Kramer Test Calculator (Free HSD) | StatsUnlock

The Tukey Kramer test is a post hoc test that compares every pair of group means after a one-way ANOVA when the groups have unequal sample sizes, while holding the overall chance of a false positive at 5%.

For each pair it divides the difference in means by a standard error built from the ANOVA error variance and both group sizes. The result, q, is compared with the studentized range distribution. With equal group sizes it gives exactly the same answer as the Tukey HSD test.

Rule of thumb: a pair differs when q is larger than the critical value q(α, k, N − k), or when the adjusted p-value is below 0.05. With 4 groups and 22 error degrees of freedom the critical value is 3.927.

The Tukey Kramer test is a multiple comparison procedure. A one-way ANOVA tells you that at least one group mean differs from the others. It does not tell you which one. The Tukey Kramer post hoc test answers that second question by testing every possible pair of groups in one step.

John Tukey designed the original method in 1949 for groups of equal size. Clyde Kramer extended it in 1956 so that it also works when groups have different numbers of observations. That extension is the only difference between the two names. Many programs still print "Tukey HSD" on output that is really the Tukey Kramer HSD test.

What it measures. For each pair it measures how many standard errors apart the two means are. That number is called q, the studentized range statistic. A large q means the gap between two means is big compared with the noise inside the groups.

A worked one-liner. Fertilizer B plants averaged 69.86 cm from 7 plots and Control plants averaged 53.00 cm from 8 plots. The ANOVA error variance was 14.198. So SE = √[(14.198 ÷ 2) × (1 ÷ 7 + 1 ÷ 8)] = 1.379, and q = 16.857 ÷ 1.379 = 12.22. The critical value is 3.927, so the two groups clearly differ.

Who uses it. Agronomists comparing treatments in field trials, clinical researchers comparing dose groups, ecologists comparing habitats, teachers comparing class sections, and engineers comparing suppliers. In all of these the groups rarely end up the same size, which is exactly when to use the Tukey Kramer test.

Tukey HSD vs Tukey Kramer vs the alternatives

4.1 What the Tukey Kramer test actually tells you

The test gives one answer per pair of groups: is the gap between these two means larger than chance alone would produce, once you allow for the fact that you looked at every pair? That last part matters. If you compare enough pairs, some gap will look big by luck. The Tukey Kramer test raises the bar so that the chance of even one false alarm across the whole set stays at your chosen level, usually 5%.

So a significant pair is a strong statement. It says this difference stood out even after the test was made stricter for all the looking you did. In the worked example, Fertilizer B beat the Control by 16.86 cm with an adjusted p below .001. You can say with confidence that B plants are taller.

4.2 How to read the mean difference and its sign

Each row shows the second group minus the first. A positive number means the second group has the higher mean. A negative number means it has the lower mean. The sign carries no judgement. It only tells you the direction. "Fertilizer C − Fertilizer B = −11.86" means C plants were 11.86 cm shorter than B plants on average.

Always read the difference in the units you measured. A difference of 11.86 cm on plants about 60 cm tall is large. The same number on trees 20 metres tall would not matter. The p-value cannot make that judgement for you.

4.3 The q statistic and the critical value

The q statistic is the difference in means divided by its standard error. It is a signal-to-noise ratio. A q of 5.91 means the two means are 5.91 standard errors apart. The critical value is the size of q you would need to call a pair different. It depends on three things: your alpha, the number of groups k, and the error degrees of freedom N − k.

More groups push the critical value up, because more pairs give chance more opportunities. With 22 error degrees of freedom the critical q is 2.93 for 2 groups, 3.55 for 3 groups, 3.93 for 4 groups and 4.20 for 5 groups. This is why adding groups you do not care about costs you power on the pairs you do care about.

4.4 Adjusted p-values and why they are larger than t-test p-values

The adjusted p-value answers this question: if all group means were truly equal, how often would the largest gap among all pairs be at least this big? Because it refers to the largest gap in the whole family, it is always bigger than the p-value from a plain t-test on the same two groups.

In the worked example, Fertilizer C against Control gives q = 3.29 and an adjusted p of .122. A simple two-sample t-test on those same two groups gives p = .066. Neither is below .05 here, but the gap between them shows the price of looking at six pairs. Never mix the two kinds of p-value in one table.

4.5 Simultaneous confidence intervals are the most useful output

Each interval gives a range of plausible values for the true difference. If the interval does not include zero, the pair is significant. The intervals are called simultaneous because all of them hold together with 95% confidence. That is a stronger promise than six separate 95% intervals, so each one is wider.

Read both ends. Fertilizer A against Control gives 8.50 cm with an interval from 2.85 to 14.15. The fertilizer helps, but the true gain might be as small as 3 cm or as large as 14 cm. If a 3 cm gain would not pay for the fertilizer, the significant result is not yet a decision.

4.6 What "not significant" does and does not mean

A pair that is not significant has not been shown to be equal. The test only says the data cannot separate the two groups. Fertilizer C against Fertilizer A gives a difference of −3.50 cm with an interval from −9.84 to 2.84. The true difference could be zero. It could also be nearly 10 cm. With only 5 and 6 plots, the study cannot tell.

Wide intervals around zero mean "we do not know". Narrow intervals around zero mean "any difference is small". Only the second one supports a claim that two groups are about the same.

4.7 Reading the compact letter display

Letters summarise the whole table in one line. Groups that share a letter are not significantly different. Groups with no letter in common are. In the example the letters are: Fertilizer B a, Fertilizer A b, Fertilizer C bc, Control c.

Fertilizer B stands alone at the top. Fertilizer C has two letters, b and c. It overlaps with Fertilizer A above it and with the Control below it, so it cannot be separated from either. Yet A and the Control do differ from each other. This pattern looks odd at first, but it is normal. It simply reflects that C sits in the middle with a small sample.

4.8 Effect sizes: how big, not only how sure

Eta squared is the share of all variation explained by the groups. In the example it is 0.779, so the fertilizer type accounts for about 78% of the variation in height. Omega squared, 0.741, is a slightly smaller and less biased version, and it is the better one to report for small samples.

Hedges g describes one pair. It is the mean difference divided by the pooled standard deviation, with a small-sample correction. By convention 0.2 is small, 0.5 is medium and 0.8 is large. Field experiments with tight control often give values far above 0.8, as this one does. That is fine. It means the groups barely overlap.

4.9 When the ANOVA and the Tukey Kramer test disagree

Sometimes the ANOVA is significant but no pair is. Sometimes a pair is significant but the ANOVA is not. Both can happen, because the two tests ask different questions. The F test looks at all means together. The Tukey Kramer test looks at the largest gap. Disagreement happens near the boundary, when p-values sit close to 0.05.

You do not need a significant ANOVA before running the Tukey Kramer test. The test controls the error rate on its own. Requiring both makes the procedure more conservative than it claims to be. If they disagree, report both honestly and treat the result as borderline.

4.10 Decision guide and common mistakes this calculator prevents

When reporting Tukey Kramer test results, readers need five things: the ANOVA result, the name of the post hoc test, why you chose it, the size and direction of each difference with its confidence interval, and the adjusted p-value. The five templates below fill themselves with your numbers after you run the analysis.

▶ Run the analysis above to auto-fill all five examples with your results.

5.1 Methods section wording

Say what was compared, which test was used and why. A good sentence is: "Group means were compared with a one-way ANOVA followed by Tukey-Kramer post hoc tests, because all pairwise comparisons were of interest and group sizes were unequal. The family-wise error rate was set at .05."

5.2 Results sentence

Lead with the ANOVA, then the pairs. Give direction in words: "Fertilizer B produced taller plants than the Control". Follow with the numbers in brackets. Do not write "there was a significant difference between B and Control" without saying which was higher.

5.3 Table and figure conventions

A results table needs one row per pair with the mean difference, standard error, q or t, adjusted p and the confidence interval. A figure should show group means with intervals and the compact letters above each bar, as chart 1 does. State in the caption that groups sharing a letter do not differ by the Tukey-Kramer test at your alpha.

5.4 Discussion sentence

Link the size of the difference to its practical meaning. "The 16.9 cm gain from Fertilizer B is about a third of the control height, which is large enough to matter commercially." Mention any pairs that were inconclusive and what sample size would resolve them.

5.5 Reporting checklist

• ANOVA: F, both degrees of freedom, p and an effect size.
• The name "Tukey-Kramer" and the reason for using it.
• The family-wise alpha.
• For each pair: difference, confidence interval, adjusted p.
• Group n, mean and SD.
• Results of the normality and equal-variance checks.
• The software used.

The Tukey Kramer test formula is built in seven steps. Each card below defines one quantity before the next card uses it.

This flowchart shows how the Tukey Kramer test is calculated step by step, from raw group data to a decision for each pair. Every box uses the worked example from this page: plant height in cm from a fertilizer trial with a Control and three fertilizers.

Check any box yourself: the Excel sheet in Section 9, the R script in Section 10 and the Python script in Section 11 all return MSE = 14.198, critical q = 3.927 and q = 12.22 for Fertilizer B against the Control.

1. Enter your data. Each group has its own box. Type or paste numbers separated by commas, exactly as the placeholder shows: 52, 48, 55, 61, 47, ... New lines, tabs, semicolons and spaces also work, so a column pasted from Excel is fine.
2. Name each group. The name field above every box is editable. Type something meaningful such as Fertilizer A. The name appears in every table, all four charts and the exported report.
3. Add or remove groups. Press Add group for more groups, up to 12. Each box has its own Clear and Remove button.
4. Or upload a file. On the Upload tab choose a CSV or Excel file, then click the column names you want. Every column you click is loaded as a separate group. Files in long format, with one column of group names and one of values, are also supported.
5. Try a sample dataset. Ten built-in datasets cover the cases that matter: a textbook example, a clear dose response, a borderline result, a null result, a small pilot, a large survey with a trivial effect and one where the assumptions fail.
6. Describe the study. The study name, what was measured and the units are optional. They are used to write full sentences in the reports.
7. Choose alpha. Leave it at 0.05 unless your field or your pre-registration says otherwise.
8. Press Run Tukey-Kramer Test. Nothing is calculated until you do. Changing any input clears the results, so you never read numbers from old data.
9. Read the assumption checks before the p-values. If Levene's test fails, the tool tells you to use Games-Howell. If it passes, read the pairwise table and chart 3.
10. Copy the write-up and export. Section 5 fills five reporting styles with your numbers. The buttons under the charts save a text report or a print-ready PDF.

Excel has no built-in function for the Tukey Kramer test, and the Analysis ToolPak does not offer it either. What Excel does well is the arithmetic: the group means, the pooled error variance, the standard error for each pair and the q statistic. The one thing Excel cannot do is look up the studentized range distribution. So you compute q in the sheet and compare it with a critical value from the table in Section 12, or from this calculator.

9.1 The functions you need

9.2 Step by step

9.3 The awkward cases

Adjusted p-values. Excel cannot compute the Tukey Kramer p-value, because it has no studentized range function. The closest built-in route is a Bonferroni p-value, which is a little larger and so a little more cautious:

The 6 is the number of pairs, and dividing q by √2 turns it into a t statistic. For Fertilizer A against the Control this gives 0.0023, where the Tukey Kramer value is 0.0021. For Fertilizer C against the Control it gives 0.1772, against 0.1222. The classic mistake is to leave out the multiplication by the number of pairs.

A critical value for degrees of freedom not in the table. Printed tables skip values. For 22 degrees of freedom, take the entries for 20 (3.958) and 24 (3.901) and interpolate, or read the exact value 3.927 from this calculator.

Many groups. With 6 groups there are 15 pairs and 15 rows to type. Lay the means out in both a row and a column and fill a square table instead. Use INDEX to pick each mean and each n so one formula fills the whole grid.

The fastest check is to paste the same columns into the calculator on this page and compare the q values. They should agree to three decimal places.

9.4 Why your Excel answer might differ from R or Python

9.5 Charting the result in Excel

1. Select the group names in row 1 and the means in row 12.
2. Choose Insert → Charts → Clustered Column.
3. Click the chart, press the + button, tick Error Bars, then choose More Options → Custom → Specify Value. Point it at a row holding =T.INV.2T(0.05,$H$3)*SQRT($H$4/A11) for each group, which is the 95% margin.
4. Add a text box above each bar with its letter from the compact letter display, and say in the caption that groups sharing a letter do not differ.
5. Remove the gridlines and the chart border, and label the y axis with the units.

Chart 1 in Section 3 of this page draws the same figure with the letters already placed, which is usually quicker than building it in Excel.

9.6 Excel errors you will probably hit

The Tukey Kramer test in R is one line: TukeyHSD(aov(value ~ group, data = df)). The function is named for Tukey HSD, but it uses the Kramer standard error whenever group sizes differ, so there is nothing extra to switch on. The one-liner prints a table and stops there. The script below also checks the assumptions, shows the calculation by hand so every number can be traced, prints a plain-English verdict and saves one 300 dpi figure.

10.1 The complete script

R, base only, saves tukey_kramer_plot.png

# ============================================================
# Tukey-Kramer test with one publication-ready figure
# StatsUnlock - https://statsunlock.com/tukey-kramer-test-calculator/
# Base R only. No packages to install.
# ============================================================

# ---- 1. Your data ------------------------------------------
# Replace this list with your own groups, or read a long-format CSV:
#   df <- read.csv("data.csv")[, c("value", "group")]
#   df <- df[complete.cases(df), ]
groups <- list(
  "Control"      = c(52, 48, 55, 61, 47, 50, 58, 53),
  "Fertilizer A" = c(60, 57, 66, 63, 59, 64),
  "Fertilizer B" = c(68, 72, 65, 70, 74, 69, 71),
  "Fertilizer C" = c(56, 62, 58, 54, 60)
)
alpha <- 0.05
units <- "cm"

df <- stack(groups)
names(df) <- c("value", "group")
df$group <- factor(df$group, levels = names(groups))   # keep your order, not alphabetical

# ---- 2. One-way ANOVA and the pooled error variance --------
fit      <- aov(value ~ group, data = df)
tab      <- summary(fit)[[1]]
k        <- nlevels(df$group)
N        <- nrow(df)
df_error <- tab[["Df"]][2]
mse      <- tab[["Mean Sq"]][2]
F_val    <- tab[["F value"]][1]
p_anova  <- tab[["Pr(>F)"]][1]
eta2     <- tab[["Sum Sq"]][1] / sum(tab[["Sum Sq"]])

# ---- 3. The built-in Tukey-Kramer test ---------------------
tk     <- TukeyHSD(fit, conf.level = 1 - alpha)$group
q_crit <- qtukey(1 - alpha, nmeans = k, df = df_error)

# ---- 4. The same thing by hand, so every number is visible -
n  <- tapply(df$value, df$group, length)
m  <- tapply(df$value, df$group, mean)
pr <- combn(levels(df$group), 2)
hand <- t(apply(pr, 2, function(p) {
  d  <- unname(m[p[2]] - m[p[1]])
  se <- unname(sqrt(mse / 2 * (1 / n[p[1]] + 1 / n[p[2]])))   # the Kramer standard error
  q  <- abs(d) / se
  c(diff = d, se = se, q = q, p = ptukey(q, k, df_error, lower.tail = FALSE))
}))
rownames(hand) <- paste(pr[2, ], pr[1, ], sep = " - ")
stopifnot(all(abs(hand[, "p"] - tk[, "p adj"]) < 1e-6))

# ---- 5. Assumption checks ----------------------------------
sw  <- shapiro.test(residuals(fit))
z   <- abs(df$value - ave(df$value, df$group, FUN = median))
lev <- anova(lm(z ~ df$group))          # Levene test, median centred

# ---- 6. Printed summary ------------------------------------
cat(sprintf("k = %d, N = %d, n per group = %s\n", k, N, paste(n, collapse = ", ")))
cat(sprintf("ANOVA: F(%d, %d) = %.4f, p = %.3g, eta squared = %.4f\n",
            k - 1, df_error, F_val, p_anova, eta2))
cat(sprintf("MSE = %.4f, critical q(%d, %d) = %.4f\n\n", mse, k, df_error, q_crit))
out <- cbind(hand[, c("diff", "se", "q")], "p adj" = tk[, "p adj"],
             lwr = tk[, "lwr"], upr = tk[, "upr"])
print(round(out, 4))
cat(sprintf("\nShapiro-Wilk on residuals: W = %.4f, p = %.4f\n", sw$statistic, sw$p.value))
cat(sprintf("Levene (median): F = %.4f, p = %.4f\n", lev[["F value"]][1], lev[["Pr(>F)"]][1]))
n_sig <- sum(tk[, "p adj"] < alpha)
ok    <- sw$p.value >= 0.05 && lev[["Pr(>F)"]][1] >= 0.05
cat(sprintf("\nVerdict: %d of %d pairs differ at family-wise alpha = %.2f. %s\n",
            n_sig, nrow(tk), alpha,
            if (ok) "Assumptions hold, so report these results."
            else "An assumption failed, so confirm with Games-Howell."))

# ---- 7. One figure: simultaneous confidence intervals ------
png("tukey_kramer_plot.png", width = 2700,
    height = round(300 * (0.62 * nrow(tk) + 2.6)), res = 300)
par(mar = c(5, 13, 5, 2), family = "sans")
ypos <- rev(seq_len(nrow(tk)))
sig  <- tk[, "p adj"] < alpha
plot(NA, xlim = range(c(tk[, "lwr"], tk[, "upr"], 0)),
     ylim = c(0.5, nrow(tk) + 0.5), yaxt = "n", bty = "n", ylab = "",
     xlab = sprintf("Difference in means (%s) with %.0f%% simultaneous CI",
                    units, 100 * (1 - alpha)))
grid(nx = NULL, ny = NA, col = "#e5e7eb", lty = 1)
abline(v = 0, lty = 2, lwd = 1.6, col = "#111827")
segments(tk[, "lwr"], ypos, tk[, "upr"], ypos, lwd = 7, lend = 1,
         col = ifelse(sig, "#7e22ce", "#94a3b8"))
points(tk[, "diff"], ypos, pch = 21, bg = "#111827", col = "white", cex = 1.5)
axis(2, at = ypos, labels = rownames(hand), las = 1, tick = FALSE, cex.axis = 0.85)
title(main = "Tukey-Kramer test: pairwise differences", adj = 0, line = 2.6, cex.main = 1.15)
title(main = sprintf("F(%d, %d) = %.2f, MSE = %.2f, critical q = %.3f",
                     k - 1, df_error, F_val, mse, q_crit),
      adj = 0, line = 1.2, cex.main = 0.95, font.main = 1)
dev.off()
cat("\nSaved tukey_kramer_plot.png\n")

Run it with Rscript tukey_kramer.R from a terminal, or paste it into the RStudio console. The figure is saved as tukey_kramer_plot.png in your working directory, which you can find with getwd(). The script contains no random numbers, so there is nothing to seed and the figure is identical on every run.

10.2 What each part does

10.3 What the figure shows

The figure has one horizontal line per pair. The dot is the difference in means and the line is its 95% simultaneous confidence interval. The dashed vertical line marks zero. Any interval that does not touch the dashed line is a significant difference, and it is drawn in purple.

For the fertilizer trial, four intervals sit clear of zero and two cross it. The widest intervals belong to pairs that include Fertilizer C, the group with only 5 plots. That is the Kramer standard error at work: smaller groups get wider intervals.

10.4 Common R problems and fixes

10.5 Useful one-liners

In Python the test is scipy.stats.tukey_hsd(a, b, c, d), available from SciPy 1.8. Like R, the function is named for Tukey HSD but applies the Kramer adjustment when the samples differ in size. The script below is complete and runnable. It computes every number by hand, confirms them against SciPy's own result, checks the assumptions, prints a verdict and saves one 300 dpi figure.

Install with pip install numpy scipy matplotlib. You need SciPy 1.8 or newer. Check with python -c "import scipy; print(scipy.__version__)".

11.1 The complete script

Python 3, saves tukey_kramer_plot.png

# ============================================================
# Tukey-Kramer test with one publication-ready figure
# StatsUnlock - https://statsunlock.com/tukey-kramer-test-calculator/
# ============================================================

import numpy as np
import matplotlib.pyplot as plt
from itertools import combinations
from scipy import stats

# ---- 1. Your data ------------------------------------------
# Replace this dictionary with your own groups, or read a CSV:
#   import pandas as pd
#   df = pd.read_csv("data.csv").dropna(subset=["group", "value"])
#   groups = {g: d["value"].to_numpy(float) for g, d in df.groupby("group")}
groups = {
    "Control":      [52, 48, 55, 61, 47, 50, 58, 53],
    "Fertilizer A": [60, 57, 66, 63, 59, 64],
    "Fertilizer B": [68, 72, 65, 70, 74, 69, 71],
    "Fertilizer C": [56, 62, 58, 54, 60],
}
alpha = 0.05
units = "cm"

names = list(groups)
data  = [np.asarray(groups[g], dtype=float) for g in names]
k     = len(data)
n     = np.array([len(x) for x in data])
N     = n.sum()
means = np.array([x.mean() for x in data])

# ---- 2. One-way ANOVA and the pooled error variance --------
F, p_anova = stats.f_oneway(*data)
df_error   = N - k
mse        = sum(((x - x.mean()) ** 2).sum() for x in data) / df_error
ss_between = sum(len(x) * (x.mean() - np.concatenate(data).mean()) ** 2 for x in data)
eta2       = ss_between / (ss_between + mse * df_error)

# ---- 3. Tukey-Kramer by hand, so every number is visible ---
q_crit = stats.studentized_range.ppf(1 - alpha, k, df_error)
rows = []
for i, j in combinations(range(k), 2):
    diff = means[j] - means[i]
    se   = np.sqrt(mse / 2 * (1 / n[i] + 1 / n[j]))   # the Kramer standard error
    q    = abs(diff) / se
    p    = stats.studentized_range.sf(q, k, df_error)
    rows.append((names[j], names[i], diff, se, q, p, diff - q_crit * se, diff + q_crit * se))

# ---- 4. Cross-check against SciPy's built-in ----------------
res = stats.tukey_hsd(*data)        # uses the Kramer form when n is unequal
for b, a, diff, se, q, p, lo, hi in rows:
    i, j = names.index(a), names.index(b)
    assert abs(res.pvalue[j, i] - p) < 1e-6

# ---- 5. Assumption checks ----------------------------------
resid       = np.concatenate([x - x.mean() for x in data])
W, p_norm   = stats.shapiro(resid)
L, p_levene = stats.levene(*data, center="median")

# ---- 6. Printed summary ------------------------------------
print(f"k = {k}, N = {N}, n per group = {n.tolist()}")
print(f"ANOVA: F({k - 1}, {df_error}) = {F:.4f}, p = {p_anova:.3g}, eta squared = {eta2:.4f}")
print(f"MSE = {mse:.4f}, critical q({k}, {df_error}) = {q_crit:.4f}\n")
print(f"{'comparison':<30}{'diff':>9}{'SE':>9}{'q':>9}{'p adj':>9}{'CI low':>9}{'CI high':>9}")
for b, a, diff, se, q, p, lo, hi in rows:
    flag = "*" if p < alpha else ""
    print(f"{b + ' - ' + a:<30}{diff:>9.4f}{se:>9.4f}{q:>9.4f}{p:>9.4f}{lo:>9.4f}{hi:>9.4f} {flag}")
print(f"\nShapiro-Wilk on residuals: W = {W:.4f}, p = {p_norm:.4f}")
print(f"Levene (median): F = {L:.4f}, p = {p_levene:.4f}")
n_sig = sum(r[5] < alpha for r in rows)
ok = p_norm >= 0.05 and p_levene >= 0.05
print(f"\nVerdict: {n_sig} of {len(rows)} pairs differ at family-wise alpha = {alpha}.",
      "Assumptions hold, so report these results." if ok
      else "An assumption failed, so confirm with Games-Howell.")

# ---- 7. One figure: simultaneous confidence intervals ------
fig, ax = plt.subplots(figsize=(8.5, 0.62 * len(rows) + 2.2))
ypos = np.arange(len(rows))[::-1]
for y, (b, a, diff, se, q, p, lo, hi) in zip(ypos, rows):
    colour = "#7e22ce" if p < alpha else "#94a3b8"
    ax.plot([lo, hi], [y, y], color=colour, lw=6, solid_capstyle="round", alpha=0.75, zorder=2)
    ax.plot(diff, y, "o", color="#111827", ms=7, mec="white", mew=1.3, zorder=3)
ax.axvline(0, color="#111827", lw=1.4, ls="--", zorder=1)
ax.set_yticks(ypos)
ax.set_yticklabels([f"{b} - {a}" for b, a, *_ in rows], fontsize=10)
ax.set_xlabel(f"Difference in means ({units}) with {100 * (1 - alpha):.0f}% simultaneous CI", fontsize=11)
ax.set_title("Tukey-Kramer test: pairwise differences\n"
             f"F({k - 1}, {df_error}) = {F:.2f}, MSE = {mse:.2f}, critical q = {q_crit:.3f}",
             fontsize=12, fontweight="bold", loc="left")
ax.grid(axis="x", alpha=0.25, zorder=0)
ax.spines[["top", "right"]].set_visible(False)
fig.savefig("tukey_kramer_plot.png", dpi=300, bbox_inches="tight")
print("\nSaved tukey_kramer_plot.png")

Run it with python tukey_kramer.py. In Jupyter, add plt.show() after the savefig line to display the figure as well. The script contains no random numbers, so there is nothing to seed.

11.2 What each part does

11.3 What the figure shows

The same figure as the R script: one line per pair, with the difference as a dot and its 95% simultaneous confidence interval as a bar. Purple bars are significant, grey bars are not, and the dashed line marks zero.

Reading it takes a few seconds. Four bars sit entirely on one side of zero. Two bars cross it: Fertilizer C against the Control, and Fertilizer C against Fertilizer A. Both involve the smallest group, and both have the widest bars.

11.4 Common Python problems and fixes

11.5 Useful one-liners

12.1 Critical values of the studentized range, α = 0.05

Conclusion: find your number of groups across the top and your error degrees of freedom, N − k, down the side. A pair differs when its q is larger than the table value. Values were computed from the studentized range distribution with SciPy and agree with published tables.

12.2 Critical values of the studentized range, α = 0.01

Conclusion: the stricter 1% level needs a q about 20% to 35% larger once there are 20 or more error degrees of freedom, and much larger with fewer. Use it when a false positive would be costly.

12.3 Why a correction is needed: error rate from uncorrected t-tests

Conclusion: with 5 groups, ten uncorrected t-tests give about a 40% chance of at least one false positive. The Tukey Kramer multiple comparison test holds that figure at 5% however many groups you have.

12.4 Effect size interpretation bands

Conclusion: these are Cohen's general conventions. Many fields see larger effects as routine, especially controlled laboratory and field experiments, so compare with published values in your own area where you can.

12.5 Which post hoc test should you use?

Conclusion: choose the Tukey Kramer test when you want every pair, the variances are similar and the groups are independent. That covers most one-way designs.

12.6 Assumption check thresholds

Conclusion: unequal variances are the serious problem, and they are most damaging when the small groups are the ones with the large variances. Unequal group sizes alone are not a problem for this test.

12.7 Smallest detectable difference by group size

Conclusion: each entry is the smallest difference between two means that would be significant at α = 0.05, in units of the within-group standard deviation, for equal groups of size n. With 4 groups of 5, two means must be about 1.8 standard deviations apart. With 4 groups of 30 the figure drops to about 0.67. Small studies can only confirm large effects.

Eight worked examples of the Tukey Kramer test, each with real numbers you can reproduce by choosing the matching sample dataset in Section 2. Every Tukey Kramer test example shows what the result looks like and how to write it up.

Study design: the Tukey Kramer test assumes a one-way design with three or more independent groups, one continuous outcome measured once on each unit, and no unit belonging to more than one group.

1. Decide that you need every pairwise comparison before collecting data. The Tukey Kramer post hoc test is built for the case where all pairs matter. If you only care about each treatment against a control, plan a Dunnett test, which needs fewer units.
2. Define the unit of observation. One plot, one patient, one student or one test piece. Several readings from the same unit must be averaged first. Treating them as separate values inflates N and produces false positives.
3. Assign units to groups at random where the design allows it. If groups are natural, such as habitats or regions, sample randomly within each.
4. Plan the sample size for the Tukey Kramer test, not for the ANOVA. Pairwise tests need more data than the overall F test. Use Table 12.7 to see what difference you could detect.
5. Aim for equal group sizes, but do not discard data to get them. Equal sizes give the most power. If you lose units along the way, keep every remaining value. Handling unequal sample sizes is what the Tukey Kramer test is for.
6. Use the same instrument, units and procedure in every group. A difference in method between groups will look like a difference between treatments.
7. Measure blind to group where a human judgement is involved.
8. Record missing values as blanks, never as zero. A zero is read as a real measurement and will pull the group mean down.
9. Write down the outlier rule in advance. One extreme value can inflate the MSE and hide every real difference.
10. Check the assumptions before reading the p-values. Plot the raw data, then look at the Shapiro-Wilk and Levene results.

The layout this calculator expects:

One column per group, one row per unit, numbers only below the header row. Blank cells are ignored, so columns can be different lengths. A long layout with a group column and a value column also works on the Upload tab.

Minimum sample: the arithmetic needs at least 2 values per group. For results worth reporting, aim for at least 5 per group, and 10 or more if you expect medium-sized effects.

Pre-registration: record the outcome, the groups, the family-wise alpha and the choice of the Tukey Kramer test before looking at the data. State what you will do if Levene's test fails, for example switch to Games-Howell.

Common collection mistakes:

• Counting repeated readings from one unit as separate observations.
• Filling short columns with zeros.
• Mixing units, such as cm in one group and mm in another.
• Dropping values to make the groups equal in size.
• Adding a group after seeing the results.

This free Tukey Kramer test calculator is designed for one question: which of several independent groups differ from which. Knowing when to use the Tukey Kramer test, and when not to, matters more than the arithmetic.

Use the Tukey Kramer test when:

• You have 3 or more independent groups and one numeric outcome.
• You want to compare every pair of groups.
• Group sizes are unequal, or equal. It works for both.
• The spread is similar in every group.
• The residuals are roughly normal, or the groups are fairly large.

Do not use it when:

• Variances are clearly unequal. Use Games-Howell.
• The same subjects appear in every group. Use a repeated-measures method.
• You only want each group against a control. Use Dunnett.
• The outcome is a rank, a count with many zeros or a yes/no result. Use Kruskal-Wallis with Dunn tests, or a model made for counts or proportions.
• You have two factors and care about their interaction. Use a two-way ANOVA with comparisons of the cell means.

Real-world examples:

1. Agriculture. Yield under four fertilizers, where some plots were lost to flooding.
2. Medicine. Blood pressure change under a placebo and three doses, with different numbers of patients finishing each arm.
3. Ecology. Species richness at point counts in four habitats, where access limited the number of counts in some.
4. Education. Exam scores under four teaching methods, in classes of different sizes.

Study design guidance: the test is most powerful with equal groups, so plan for balance. But a design that ends up unbalanced is not a damaged design. Keep all the data and let the Kramer standard error do its job.

Decision tree:

Assumptions

1. Independent observations. Each value comes from a different unit, and units in one group are unrelated to units in another. Check: from the study design, since no test on the numbers can show it. If it fails: the standard errors are too small, so p-values are too small and false positives rise.
2. Normal residuals. Within each group the values follow a roughly normal distribution. Check: the Shapiro-Wilk test on the residuals, and chart 2. If it fails: with heavy tails or outliers the MSE is inflated, so the test loses power and misses real differences. With strong skew and small groups the error rate can go either way.
3. Equal variances. Every group has the same underlying spread. Check: the Levene test and the variance ratio. If it fails: when small groups have the large variances, p-values are too small and false positives rise. When large groups have the large variances, p-values are too large and power falls.
4. A continuous outcome. The measurement is on an interval or ratio scale. Check: by inspection. If it fails: means of ranks or categories have no clear meaning, so the conclusions cannot be trusted in either direction.

Limitations

• It is slightly conservative with unequal n. The true family-wise error rate is a little below alpha, so power is a little lower than the nominal level. No exact alternative of similar simplicity exists, and the loss is small.
• It tests pairs only. It cannot test a contrast such as the average of two groups against a third. Use the Scheffé method for that.
• It loses power as groups are added. Every extra group raises the critical value. If only comparisons with a control matter, use the Dunnett test.
• It is not robust to unequal variances. Use the Games-Howell test.
• It is not robust to outliers. Use the Kruskal-Wallis test with Dunn comparisons, or a trimmed-means method.
• It does not handle covariates or more than one factor. Use ANCOVA or a factorial model with estimated marginal means.
• This calculator handles up to 12 groups. For more, use the R or Python script from Sections 10 and 11.

The Tukey Kramer test answers the question an ANOVA leaves open: which groups differ. It compares every pair of means, it works whether the groups are equal or unequal in size, and it keeps the chance of even one false positive across the whole set of comparisons at the level you choose.

Three habits make the difference between a correct analysis and a misleading one. First, check the equal-variance assumption before reading any p-value, because that is the assumption the test cannot survive. Second, read the confidence intervals and not only the significance labels, because an interval tells you how large the difference might be and a label does not. Third, report every pair, including the ones that did not reach significance, and never describe an inconclusive pair as "no difference".

The method is the same in every program. R's TukeyHSD, SciPy's tukey_hsd, SPSS, JMP and this calculator all compute q from the pooled error variance and both group sizes, then refer it to the studentized range distribution. Excel can do all of the arithmetic but not that last step, which is why a table or a calculator is needed there.

Paste your groups into the calculator above, read the assumption checks first, then use chart 3 and the write-up in Section 5 to report your Tukey Kramer test.

Adjusted p-value

A p-value that already allows for the number of comparisons made. Compare it directly with alpha.

Alpha

The chance of a false positive you are willing to accept, usually 0.05.

ANOVA

Analysis of variance. A test of whether three or more group means are all equal.

Compact letter display

Letters placed beside group means. Groups that share a letter are not significantly different.

Confidence interval

A range of plausible values for a true difference, built from the data.

Critical value

The size a test statistic must exceed to be called significant.

Degrees of freedom

The number of independent pieces of information used to estimate a variance. Here it is N minus k.

Effect size

A measure of how large a difference is, separate from how certain it is.

Eta squared

The share of total variation explained by the groups, from 0 to 1.

Family-wise error rate

The chance of at least one false positive across a whole set of comparisons.

Games-Howell test

A post hoc test for all pairs that does not assume equal variances.

Hedges g

The difference between two means divided by the pooled standard deviation, corrected for small samples.

HSD

Honestly significant difference. The smallest gap between two means that the Tukey test calls significant.

Levene test

A test of whether groups have equal variances.

MSE

Mean square error. The pooled variance inside the groups, taken from the ANOVA table.

Multiple comparisons

Running several tests on the same data, which raises the chance of a false positive unless corrected.

Omega squared

A version of eta squared with less upward bias in small samples.

Post hoc test

A test run after an ANOVA to find which groups differ.

Residual

An observation minus the mean of its own group.

Shapiro-Wilk test

A test of whether data follow a normal distribution.

Simultaneous confidence intervals

A set of intervals that all contain their true values together with the stated confidence.

Standard error

The expected amount by which an estimate, such as a difference in means, varies from sample to sample.

Studentized range

The gap between the largest and smallest of several means, divided by a standard error. Its distribution gives the Tukey critical values.

Tukey-Kramer test

The Tukey HSD test with a standard error that allows for unequal group sizes.

The Tukey Kramer test calculator on this page follows the methods in these 20 peer-reviewed journal articles. Each entry links to the publisher through its DOI.

1. Tukey, J. W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99–114. https://doi.org/10.2307/3001913
2. Kramer, C. Y. (1956). Extension of multiple range tests to group means with unequal numbers of replications. Biometrics, 12(3), 307–310. https://doi.org/10.2307/3001469
3. Hayter, A. J. (1984). A proof of the conjecture that the Tukey-Kramer multiple comparisons procedure is conservative. The Annals of Statistics, 12(1), 61–75. https://doi.org/10.1214/aos/1176346392
4. Dunnett, C. W. (1980). Pairwise multiple comparisons in the homogeneous variance, unequal sample size case. Journal of the American Statistical Association, 75(372), 789–795. https://doi.org/10.1080/01621459.1980.10477551
5. Dunnett, C. W. (1980). Pairwise multiple comparisons in the unequal variance case. Journal of the American Statistical Association, 75(372), 796–800. https://doi.org/10.1080/01621459.1980.10477552
6. Stoline, M. R. (1981). The status of multiple comparisons: Simultaneous estimation of all pairwise comparisons in one-way ANOVA designs. The American Statistician, 35(3), 134–141. https://doi.org/10.1080/00031305.1981.10479331
7. Games, P. A., & Howell, J. F. (1976). Pairwise multiple comparison procedures with unequal n’s and/or variances: A Monte Carlo study. Journal of Educational Statistics, 1(2), 113–125. https://doi.org/10.3102/10769986001002113
8. Copenhaver, M. D., & Holland, B. (1988). Computation of the distribution of the maximum studentized range statistic with application to multiple significance testing of simple effects. Journal of Statistical Computation and Simulation, 30(1), 1–15. https://doi.org/10.1080/00949658808811082
9. Piepho, H.-P. (2004). An algorithm for a letter-based representation of all-pairwise comparisons. Journal of Computational and Graphical Statistics, 13(2), 456–466. https://doi.org/10.1198/1061860043515
10. Day, R. W., & Quinn, G. P. (1989). Comparisons of treatments after an analysis of variance in ecology. Ecological Monographs, 59(4), 433–463. https://doi.org/10.2307/1943075
11. Midway, S., Robertson, M., Flinn, S., & Kaller, M. (2020). Comparing multiple comparisons: Practical guidance for choosing the best multiple comparisons test. PeerJ, 8, e10387. https://doi.org/10.7717/peerj.10387
12. Ruxton, G. D., & Beauchamp, G. (2008). Time for some a priori thinking about post hoc testing. Behavioral Ecology, 19(3), 690–693. https://doi.org/10.1093/beheco/arn020
13. Dunnett, C. W. (1955). A multiple comparison procedure for comparing several treatments with a control. Journal of the American Statistical Association, 50(272), 1096–1121. https://doi.org/10.1080/01621459.1955.10501294
14. Scheffé, H. (1953). A method for judging all contrasts in the analysis of variance. Biometrika, 40(1–2), 87–110. https://doi.org/10.1093/biomet/40.1-2.87
15. Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52–64. https://doi.org/10.1080/01621459.1961.10482090
16. Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38(3–4), 330–336. https://doi.org/10.1093/biomet/38.3-4.330
17. Brown, M. B., & Forsythe, A. B. (1974). Robust tests for the equality of variances. Journal of the American Statistical Association, 69(346), 364–367. https://doi.org/10.1080/01621459.1974.10482955
18. Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3–4), 591–611. https://doi.org/10.1093/biomet/52.3-4.591
19. Royston, P. (1995). Remark AS R94: A remark on Algorithm AS 181: The W-test for normality. Journal of the Royal Statistical Society: Series C (Applied Statistics), 44(4), 547–551. https://doi.org/10.2307/2986146
20. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863