Multilevel analysis quantifies variation in the experimental effect while optimizing power and preventing false positives

Table 2 Parameter settings used to generate the simulated datasets

	Variation in intercept (ICC)	Study	Aim of study	\(\sigma_{u1}^{2}\)	ICC	d	N	nc
Variation in experimental effect (\({\sigma}_{{{u}1}}^{2}\))
Absent	Absent	1a	Statistical power	0.00	0.00	0.20	10	5–50
	Absent	1a	Statistical power	0.00	0.00	0.50	30	5–50
	Present	1b	Statistical power	0.00	0.25	0.20	10	5–50
	Present	1b	Statistical power	0.00	0.50	0.50	30	5–50
Present	Absent	2a	False positive rate	0.025	0.00	0.00	50	5–105
				0.050
				0.150
	Present	2b	False positive rate	0.025	0.50	0.00	50	5–105
				0.050
				0.150

The simulations with no cluster-related variation in the experimental effect (studies 1a and 1b; \(\sigma_{u1}^{2}\) = 0) investigate the effect on the statistical power to detect the experimental effect in case that variation in the intercept is present but not accommodated. Hence, in studies 1a and 1b, the magnitude of the overall effect of the experimental manipulation, expressed by effect size d, exceeds zero (d > 0). The simulations including cluster-related variation in the experimental effect (studies 2a and 2b; \(\sigma_{u1}^{2}\) > 0) investigate the effect on the false positive rate in case that this variation in the experimental effect is not accommodated in the statistical model. Hence, in studies 2a and 2b, the magnitude of the overall effect of the experimental manipulation equals zero (d = 0)
ICC intracluster correlation, denoting the extent of dependency in the data, N number of clusters, nc number of observations per experimental condition per cluster

ISSN: 1471-2202