By now you have more machinery than any single problem needs, and the skill that actually matters is no longer running a test — software runs tests instantly, and it will run the wrong one for you just as fast. The skill is looking at a study and recognising which test the design has already chosen. Most of that decision is settled before a single number is computed: what you measured on each subject, and whether the two columns are linked subject by subject. Only the last question is about the numbers themselves, and even that one can be answered by looking. Below, seven real studies come through the door. For each you answer the tree’s questions, watch your path light up, and then see the test at the end of your path run on the actual data — side by side with the test you would have reached by answering differently. The two p‑values are never the same. Sometimes they are not even on the same side of 0.05.
The decision tree
Three questions, seven destinations. The first two are about the study, not the data — no amount of staring at the numbers will tell you whether the same person appears in both columns. Only the third question is answered by looking at what you collected, and the panels below hand you the evidence for it. Your path lights up as you answer; if it ends somewhere the design does not support, the tree will say so rather than quietly returning a number.
the path your answers tookwhere the design actually leadsbranches not taken
The data in front of you
Two shapes, one question
This is the only branch of the tree the numbers themselves decide, so it is worth settling with a simulation rather than a rule of thumb. Both groups below are drawn from the same shape; group B is simply the whole distribution multiplied by a constant, so B really does tend to sit higher whenever the shift is above zero. Slide the skew up and the shape stops being a bell and grows a long right tail — the shape of waiting times, hospital charges, insect counts, income. The dots underneath are one actual sample of each group at your current settings.
group Agroup B, shifted up
How often each test finds it
Each run below is a whole study: draw both groups, run both tests on the identical numbers, record whether each returned p < 0.05. The bars are the share of studies in which each test declared a difference. When the shift is set to zero there is nothing to find, so every bar you see is a false alarm rate and both should hover near the dashed 5% mark. Once you raise the shift, the same bars become power, and the gap between them is what the third question of the tree is worth.
two-sample t (Welch)Mann–Whitney Uthe 0.05 bin
Which test?pick a study to begin
A study comes through the door
Seven designs, one per destination on the tree. Read the description before you touch a button — the first two answers are hiding in the sentence, not in the spreadsheet.
Walk the tree
The test at the end of your path, computed live
Nothing below is looked up. Every statistic is computed from the numbers in the panel on the left, and the p‑values come from the t, normal, chi‑square and hypergeometric distributions evaluated here in the page. Read the two blocks against each other: the arithmetic is honest in both, and only one of them is answering the question the study asked.
Say it right
The truth you are sampling from
The tree’s last question — roughly normal, or skewed and ordinal? — sounds like a matter of taste. It is not. Set the world below, then run thousands of studies in it and watch which test actually finds the difference. Everything is generated from a seeded pseudo‑random stream, so the same settings always give the same answer.
The verdict of six thousand studies
Nothing run yetPress Run 1,000 studies under the charts on the left. Each study draws both groups fresh, runs the two-sample t‑test and the Mann–Whitney rank test on the same numbers, and records both p‑values.
Two honest cautions about this comparison. The rank test does not test whether the means differ; it tests whether values in one group tend to sit above values in the other. When the two shapes are the same and one is simply shifted, as here, the two questions point the same way — but with differently shaped groups they can disagree, and then you must say which one you meant. And the p‑value for Mann–Whitney is computed here from its normal approximation with a continuity correction, which is why the sample size slider does not go below ten.
What to carry out of this section
Two of the three questions are answered by the methods section, not the data. What was measured on each subject, and whether the same subject appears in both columns, are facts about how the study was run. A spreadsheet of numbers cannot tell you either one.
The software will never object. Hand paired data to an independent test and you get a p‑value. Hand a 2 × 2 table with a cell of one to a chi‑square and you get a p‑value. Choosing the test is your job precisely because nothing downstream checks it.
Ordinal is not numeric. A rating scale gives you order without equal spacing, so the mean of those ratings is not a defensible quantity even when the arithmetic runs. Rank-based tests ask a question the scale can actually support.
Exact when small, approximate when large. Fisher’s exact test and the exact binomial version of McNemar’s test enumerate the possibilities instead of leaning on a normal curve. That matters exactly when the counts are too thin for the curve to be trustworthy.
A small p‑value is a statement about data, not about the truth of a hypothesis. It is the probability of a result at least this extreme if nothing were going on. It is never the probability that nothing is going on, and it never upgrades an observational comparison into a cause.