← Chapter 13
Statistics · Chapter 13

Test of Independence

Now there are two categorical questions on the same survey form, and the thing you want to know is whether the answers travel together. Do people who commute by bicycle rate their commute differently from people who drive? You cannot answer that with an average, and you cannot answer it by comparing one number to another — the data are a whole table of counts, and you have to judge the table as a unit. The move that makes it possible is quietly ingenious: assume for a moment the two questions have nothing to do with each other, and ask what the table would have looked like. Independence pins down every expected count exactly, using nothing but the row and column totals you already have. Then you compare, cell by cell, and add up the misses the same way you did for goodness of fit. Below you set how strongly the two variables are really linked, survey real people, and watch the expected table build itself out of the margins.

One survey, cross-tabulated

Each bar is one commute mode, split by how those particular people rated their commute. Because every bar is stretched to the same length, you are looking at conditional proportions within the row — the profile of answers given that someone commutes that way. The bottom bar is everyone pooled together, and the dashed lines drop down from its boundaries. Here is the whole idea of independence in one picture: if the two questions are unrelated, every bar has the same striping, and every boundary lands on a dashed line. Press Survey new people a few times with the association set to zero — the boundaries jitter around the lines but never drift systematically away from them.

Where the expected counts come from

This is the step the whole test turns on, and it is worth slowing down for. If the two questions really are unrelated, then the chance a person falls in row i and column j is just the chance of row i times the chance of column j. We do not know those chances, so we use the best estimates on the table — the margins themselves, Ri / n and Cj / n. Multiply them together for the joint proportion, then multiply by n to turn a proportion back into a count, and the n's collapse: E = Ri × Cj ÷ n. Tap any cell to watch its row total and column total light up and see the arithmetic written out.
observed count O expected under independence E the two margins feeding the tapped cell

Building χ², cell by cell

Exactly the arithmetic from the previous section, now run over every cell of the table: square the miss so that too‑many and too‑few both count as wrong, divide by the expected count so a miss is judged against the scale it happened on, and add. The grid shades each cell by how much it contributes and colours it by direction — red where more people landed there than independence predicted, blue where fewer. In a genuine association the reds and blues form a pattern, not a scatter. Tap any row of the table to isolate its arithmetic.
CellObserved
O
Expected
E
O − E(O−E)² ÷ EAdjusted
residual

How surprising is that total?

χ² is zero only if every cell lands exactly on its expectation, and any departure in any direction pushes it up, so once again the entire question lives in the upper tail. The curve is the distribution the statistic follows when the two variables are independent — and note the degrees of freedom have changed. It is no longer “cells minus one.” It is df = (r − 1)(c − 1), because this time the expected counts were built out of the data's own margins, and every margin you borrow costs you a degree of freedom.
χ² if the two were independent this big or bigger — the p‑value your total reject beyond here

Run the whole survey a thousand times

One survey settles nothing. Each run below recruits a fresh set of people from the true joint distribution, cross‑tabulates them, rebuilds the expected counts from that table's margins, and keeps only χ². The bars are those simulated totals. The dark curve is the theoretical χ² distribution with (r − 1)(c − 1) degrees of freedom, drawn from the formula and fitted to nothing at all. Set the association to zero and the bars settle onto the curve — that agreement is the entire justification for reading a p‑value off it. Dial the association up and the pile marches right, which is what power looks like.
simulated surveys theory, not fitted rejection cut‑off
Surveys
run
0
Mean χ²
observed
—
Mean χ²
predicted
—
Share
rejecting
—
Two questions on one formset the study up, then start surveying

The survey, and the truth behind it

A city asks commuters two questions: how do you get to work, and how would you describe the trip. In real life you would know only the answers; here you also set the truth behind them, so you can watch the machinery succeed and fail against something you already know.
The table
3
3
300
The hidden truth
12
Jump to a situation worth seeing

This survey, computed live

People
counted n
—counts, never %
Table
size
—rows × columns
Degrees of
freedom
—(r−1)(c−1)
Smallest
expected E
— 
χ²
statistic
—Σ (O−E)²/E
Reject
beyond
—α = 0.05
p‑value
upper tail
— 
Cramér's
V
— 
Watch χ² and V disagree. Leave the association slider alone and drag the sample size from end to end: χ² and the p‑value swing violently, because χ² is very nearly proportional to n. Cramér's V barely moves, because it is χ² with the sample size divided back out — V = √(χ² ÷ (n × min(r−1, c−1))). One of those two numbers is telling you about the world; the other is telling you about your budget.

Is the curve even legal here?

The χ² curve is an approximation to the true behaviour of a discrete statistic, and it only becomes a good one when the expected counts are not too thin. The conservative working rule is that every expected count should be at least 5. The more permissive rule most software uses, due to Cochran, is that no expected count falls below 1 and no more than 20% of the cells fall below 5. Note carefully what the rule is about: E, not O. Observing zero people in a cell is perfectly legal. Expecting half a person is not.
Smallest
expected
—
Cells with
E < 5
—
Cochran
rule
—
n needed to
clear E ≥ 5
—

Say it right

Eleven things people say after a test of independence. Five of them are correct. Tap each one.

What to carry out of this section

E = row total × column total ÷ n, and the formula is not a rule to memorise — it is a sentence. Independence means the joint chance factors: P(row and column) = P(row) × P(column). Estimate each piece from the margins, Ri / n and Cj / n, multiply them for the joint proportion, then multiply by n to convert a proportion into a headcount. The n's cancel down to RiCj / n. If you can say that sentence you never need to look the formula up, and you will also see instantly why the expected table has exactly the same margins as the observed one.
Expected counts are almost never whole numbers, and that is fine. E = 47×62÷300 = 9.71 does not mean 9.71 people. It is the long‑run average number of people who would land in that cell across many surveys of this size if the two variables were unrelated. Rounding it before computing the statistic is a small, needless error. Only the observed counts are forced to be integers — they are people.
df = (r − 1)(c − 1), and the bookkeeping is exact. There are rc cells. Subtract 1 because the counts must total n. Subtract another (r − 1) for the row proportions you estimated from the data, and (c − 1) for the column proportions. That leaves rc − 1 − (r−1) − (c−1) = rc − r − c + 1 = (r − 1)(c − 1). Compare with goodness of fit, where the claimed proportions arrived from outside the data and cost you nothing, so df was simply k − 1. Estimating things from your own data is not free. Drag r and c and watch the curve reshape underneath a table whose story has not changed.
Independence has a picture, and it is the first panel. “The two variables are independent” is the same statement as “the conditional distribution of ratings is identical for every commute mode” — every bar striped the same way. That equivalence is worth more than the formula, because it tells you what to look at before you compute anything, and it tells you what a significant result actually claims: somewhere, some row's profile differs from the others.
Association is not causation, and this design cannot rescue you. Suppose the test comes back overwhelming and the people on foot rate their commutes far more pleasantly. Three explanations remain wide open. Walking may genuinely make the trip nicer. Or people who already live somewhere pleasant to walk chose to walk — the exposure is self‑selected, and the causal arrow runs backwards. Or something behind both, like living downtown, drives the mode and the mood independently — a confounder. The χ² test cannot distinguish these, because it only ever saw a table of counts. It certifies that the pattern is more than noise. It says nothing whatsoever about where the pattern came from.
Significant and strong are different words. Load the Faint, huge survey preset. The p‑value is minuscule and Cramér's V is close to zero, and both are correct: the association is real and it is negligible. Because χ² grows roughly in proportion to n, any association that is not exactly zero becomes significant eventually, and with enough people you will reject independence over a difference nobody could act on. Always report an effect size next to the p‑value. A p‑value answers “could this be nothing?” It never answers “how big is it?”
The p‑value is not the probability that the variables are independent. It is computed by assuming independence and asking how often a table this discrepant, or worse, would arise from pure sampling noise. It is P(data this extreme | independence), not P(independence | data). Those two are not interchangeable, and swapping them is the single most common misstatement in applied statistics. It also explains why we never “accept” the null: a large p‑value means your data were compatible with independence, and small data are compatible with almost everything.
A significant χ² does not name the guilty cell. This is an omnibus test — one number for the entire table. The natural follow‑up is the adjusted residuals in the third panel: values beyond roughly ±2 mark cells pulling hardest, and their signs show the direction. But scanning rc cells after a significant result and reporting whichever ones look big is a fishing expedition, and any cell picked that way deserves confirmation in fresh data rather than a p‑value inherited from the omnibus test.
When the expected counts are too thin, stop — do not round up and continue. Load the Strong, tiny survey preset and look at the smallest expected count. With E's below 5, the χ² curve is a poor stand‑in for the discrete truth and the p‑value cannot be trusted in either direction. The real fixes are to collapse categories that genuinely belong together, gather more people, or use Fisher's exact test, which computes the answer from the exact distribution of tables with those margins instead of leaning on a curve.
Two different studies, one identical calculation. If you sample n people at random and record both answers, you are testing independence. If instead you deliberately recruit 150 drivers, 150 cyclists, and 150 walkers and ask each group one question, the row totals were fixed by you, not by chance, and you are testing homogeneity — whether several populations share one distribution. The expected counts, the statistic, the degrees of freedom and the table you read are all the same. What differs is the sentence you are entitled to write at the end, and which margins were yours to choose.
One last thing worth doing. Set the association to zero, so the two questions are genuinely unrelated, and run five thousand surveys. The mean χ² settles on (r − 1)(c − 1) — not approximately, and not by luck — and about one survey in twenty crosses the rejection line anyway. Those tables contain no association at all. They are ordinary surveys of an ordinary city, and they are the advertised, agreed‑upon price of drawing the line at 5%.