Fig. 2

Results of Experiment 1 in which total error and FDP control were compared using several statistical testing methods. Upper panels show receiver operating characteristic (ROC) curves, where horizontal and vertical axes denote the specificity and sensitivity of the statistical test, respectively. Red and blue lines show the EB-regularized-LR (with regularization) and EB-LR (without regularization) results, respectively. In the title of each column, observation length T and the area under the ROC curve (AUC) scores of the two methods are shown. Lower panels display false positive control, where horizontal and vertical axes denote q value threshold and false discovery proportion (FDP) when the q value threshold on the horizontal axis was used, respectively. Because the q value threshold is regarded as an estimation of the FDP value, when FDP is located under and over the thin black diagonal line of \(x=y\), the corresponding q value estimations are considered conservative and aggressive, respectively. A conservative line is preferable, because by definition, the q value should be a conservative estimation of the FDP value. If the line is close to the diagonal line, the q value estimation is faithful. Blue broken, dotted, and solid lines represent the q values estimated by CHI2-LR, DOF-LR, and EB-LR for the likelihood-ratio statistics without regularization (in fact, a small regularization was applied to avoid optimization divergence, see “Shape-related statistics” section, respectively. The red line (EB-regularized-LR, proposed method) is the likelihood-ratio statistic with larger regularization. The six panels from the leftmost to the rightmost columns correspond to the experimental settings of observation lengths \(T=1000, 2000, 5000, 10{,}000, 20{,}000,\) and 50,000