Sampling error and the law of large numbers
The law of large numbers
As the number of independent repetitions grows, the observed relative frequency of an event approaches its true probability. The word “approaches” hides a lot of wobble: with only a few repetitions the observed frequency can be far from the truth, and the wobble shrinks only gradually.
Standard error
If an event has probability p and you estimate p from m independent trials, your estimate varies from experiment to experiment with a standard deviation called the standard error.
To halve the standard error you must quadruple m. Going from 100 to 10,000 samples makes estimates ten times more precise, not a hundred times.
SE = √( p (1 − p) / m ) p = 0.34, m = 1,000: SE ≈ 1.5 percentage pointsConfidence intervals
A 95% confidence interval is produced by a method that, over many repetitions of the experiment, contains the true value about 95% of the time. It is a statement about the method, not a claim that this particular interval has a 95% chance of being right once it has been computed.
The Wilson score interval used in this lab is a robust choice for proportions, especially when counts are small or probabilities are near 0 or 1.
Bias is different from noise
Sampling error is random noise that shrinks with more data. Bias is a systematic tilt, for example a flawed shuffle or an unrepresentative sample, and it does not go away however many samples you take. Good experiments reduce both: large samples for noise, careful design for bias.
Check your understanding
An estimate from 400 trials has standard error 2.5 points. What is the standard error from 1,600 trials?
1.25 points: four times the data halves the error.
Is a single run of 100 draws that is 5 points off the truth evidence of a faulty simulator?
Not necessarily. For p near 0.34 the standard error at m = 100 is about 4.7 points, so such deviations are ordinary.
Can a 95% interval miss the true value?
Yes. By design about 1 in 20 intervals will miss.