Independence: With and Without Replacement

Two draws, branch by branch

How fast the dependence dies

with replacement — always the same without replacement — the two branches how far apart they are

A real run of samples

with replacement without replacement exact probability

Independenceput the chip back, or keep it out

Box holds—

Marked—

Sample—

Send the box to

Repeats—

P(marked on draw 1)——

P(marked on 2 | marked on 1)——

P(marked on both)——

P(1st) × P(2nd)——

The two draws, side by side

Question about draw 2	Put it back	Keep it out
After a marked first draw	—	—
After a plain first draw	—	—
Distance between those two	—	—
Marked on draw 2, not told the first	—	—
Marked on both draws	—	—

Look hard at the fourth row. Even without replacement, the second draw is marked with probability exactly the same as the first — the two columns agree. Nothing about the second card is worse off until somebody tells you what the first one was. Dependence lives in the conditional rows above it, not in the unconditional one.

Counting marked items in each sample

How the sample was taken	Mean marked	Spread (SD)
Put it back — exact	—	—
Put it back — this run	—	—
Keep it out — exact	—	—
Keep it out — this run	—	—

There is one number worth carrying out of this whole page, and it is startlingly simple. Without replacement, the gap between “marked on draw 2 given the first was marked” and “marked on draw 2 given the first was plain” is exactly 1/(N − 1) — where N is how many items were in the box to begin with. The two conditionals are K/(N−1) and (K−1)/(N−1), and subtracting them leaves 1/(N−1) with the K gone entirely. How many were marked does not matter. One heart in a deck or twenty-six of them, removing a card shifts the odds by the same 1/51. That single fact is the whole of this section in miniature: dependence created by sampling is a story about the size of the box, not about what is in it. A box of 10 and the shift is 11.1 percentage points, which will wreck any calculation that ignores it. A box of 5,000 and the shift is 0.02 percentage points, which no experiment you will ever run could detect. That is why survey statisticians treat a sample of a thousand people out of a city of a million as if the draws were independent — not because they are, but because the error from pretending is smaller than the ink used to write it down.

Six ways this gets misread. First, independent does not mean unrelated in the world. It is arithmetic: A and B are independent exactly when P(A and B) = P(A) × P(B). Suit and rank in a deck are independent in this precise sense even though they are printed on the same card, and two events can be causally tangled yet still satisfy the equation. Check the multiplication; do not consult your intuition about the subject matter. Second, independent is not the same as mutually exclusive — the two are nearly opposites. Mutually exclusive means they cannot both happen, so learning that A occurred tells you B definitely did not, which is a great deal of information. Two events with non-zero probability can never be both mutually exclusive and independent. Third, the gambler’s fallacy is only a fallacy when you replace. A coin, a die, a chip put back — these genuinely have no memory, and four marked draws in a row change nothing about the fifth. But when you do not replace, the box really is depleted, and expecting the next draw to be different is not superstition, it is correct. Card counting is legal arithmetic, not a losing streak talking. Fourth, sampling without replacement does not bias anything. Both tables above show the same mean, every time; what shrinks is the spread, by the factor (N−n)/(N−1) under the square root. Drawing without replacement is more precise, not less, because a sample that cannot repeat itself covers more of the population. Fifth, the unconditional probability of the second draw never moves. Students routinely say the second card is “less likely” to be a heart. It is not: it is 13/52 either way, because before you look at the first card every position in the deck is symmetric. Only the conditional probability moves. Sixth, with three or more events, pairwise independence is not enough. Full independence requires the multiplication rule to hold for every subset at once, and it is possible to build three events that pass every pairwise test and fail the three-way one.