Almost everyone who reports a confidence interval says the wrong sentence about it. The tempting reading — “there is a 95% chance the true value is in here” — puts the randomness in the wrong place. Once the sample is drawn and the arithmetic is done, your interval is two fixed numbers and the parameter is one fixed number; the parameter is either inside or it is not, and no probability is left over. The 95% describes the recipe, not the result. Below, the recipe is run one hundred times on one hundred fresh samples from a population whose true mean we are allowed to see. Every horizontal line is a whole study someone could have run. Count the green ones. Then remember that in real work you get exactly one line, and you never learn its color.
One hundred studies, one hundred intervals
captured μmissed μx̄ — that study’s sample meanμ — the true mean
The long run the promise is about
Coverage after every interval ever built with the current recipe, on a log scale so you can watch the early chaos settle. The first few intervals swing between 0% and 100%; a hundred of them still wobble by several points; ten thousand of them sit on the line. That line is what the confidence level is a promise about — and nothing else.
Should you have gotten exactly 95?
No — and this is the exact distribution of how many out of 100 you should expect, computed from real binomial probabilities rather than simulated. The dark bar is the batch currently drawn above. Getting 92 or 97 is not a sign of anything at all.
Resting heart ratepress “Draw 100 new studies” to begin
The recipe you are repeating
What is being measured — μ is visible here only because this is a simulation
Confidence level — sets the multiplier z*
How each sample is collected — part of the recipe too
40
Multiplier z*1.960
Std. error σ/√n—
Margin of error—
Interval width—
Captured in this batch—out of 100
Intervals built so far0
Coverage so far—
What this recipe truly delivers—
The one study you actually run
Tap any line in the stack, or take a blind one. The pale dots are that study’s individual measurements; the thick bar underneath is its confidence interval. Notice how small the interval is next to the spread of the data — it is estimating the mean, not describing the individuals.
Which of these sentences are true?
Every sentence below is about the interval currently selected. Tap each one before you read the verdict.
Say it this way and you will never be wrong: “This interval came from a procedure that captures μ 95% of the time.” The confidence sits on the procedure, and the procedure is the whole thing — how the sample was drawn, how big it was, which multiplier you used, which formula produced the standard error. Run that procedure a million times and 95% of the intervals it hands you will contain μ. You get one of them. It is a good bet, and it is not a probability about your particular answer, in the same way that a surgeon with a 95% success rate does not give your completed operation a 95% chance of having gone well — it either did or it did not.
Five things the 95% is not.(1) It is not a probability about your interval. After the arithmetic, nothing random remains: μ is a fixed number and your two endpoints are fixed numbers. Probability had its moment before the sample was drawn, and it was spent. (2) It is not a statement about individuals. An interval of (69.6, 75.2) beats per minute does not mean 95% of people fall in that range — it is a claim about one number, the population mean. The middle 95% of individuals is √n times wider, and it does not shrink when you collect more data; the interval does. (3) It does not cover bias. The margin of error is built entirely from σ/√n, which describes random sampling wobble and absolutely nothing else. Switch the sampling frame above to volunteers-only and watch coverage collapse while the label still reads 95% — and watch it get worse as n grows, because a bigger biased sample is just a more confident wrong answer. Non-response, a leading question, a broken instrument, measuring the wrong people: none of it is anywhere in the formula. (4) A miss is not a mistake. Five in a hundred are supposed to miss, and the coral lines above came from exactly the same flawless recipe as the green ones. If nothing ever missed, the level would be wrong. (5) Higher confidence is not higher accuracy. Moving from 95% to 99% does not improve the estimate by one decimal — x̄ does not budge. It only widens the net. Certainty and precision trade against each other, and the only way to buy both is a larger n.