Fig. 4

Ignoring variation in the experimental effect results in inflated false positive (i.e., Type I error) rate. Inflation of the Type I error rate already occurs when a small amount of variation in the experimental effect (e.g., \(\sigma_{u1}^{2}\) = 0.025) remains unaccounted for in the statistical model, and occurs both when the intercept (i.e., mean value of the control condition) is invariant over clusters (a; ICC = 0; study 2a), and when the intercept varies substantially over clusters (b; ICC = 0.50; study 2b). In panel a, the lines depicting conventional analysis (i.e., t test on individual observations) and misspecified multilevel analysis completely overlap. Using a paired t test on the experimental condition specific cluster means results in a correct Type I error rate. In panel b, the lines depicting the paired t test and the correctly specified multilevel analysis completely overlap