Parameter optimization of logistic regression classifiers
© Sherwin and Chartier; licensee BioMed Central Ltd. 2013
Published: 8 July 2013
Logistic regression (LR) classifiers have been used successfully in the single-trial analysis of EEG data, especially in tasks of perceptual decision-making [1, 2], but heuristics govern the choices for classifier parameters, such as window size (δ). Furthermore, no rigorous definition exists as to the number of epochs (N) of either class that would allow sufficient classifier training before testing using leave-one-out cross-validation. Here, we attempt to address these issues by exploring this discrete parameter space with the aid of a genetic algorithm. In doing so, we draw preliminary conclusions on both subject-specific and subject-general trends of these classifiers.
To establish a baseline for comparison, we utilize EEG data from a previous study using LR to classify neural response to a two-choice forced-decision face vs. car visual task . In this study, a window size (δ) of 60 ms was used to segment epochs for classification. Other studies using this technique also employ a comparable window size [2, 3], even though δ has the potential to drastically affect classifier training and performance. Similarly, the number of epochs used to train the classifier can greatly affect its performance, a number too low causing an insufficient number of points through which a dividing hyperplane can be found.
Recognizing the dependence of classifier performance on these discrete parameters, we use a genetic algorithm to explore the δ vs. N design space. In doing so, we track an objective function whose value depends on maximizing an epoch window's leave-one-out Az (area under receiver-operating characteristic) value while decreasing its variability (determined from bootstrapping), which increases with a low number of epochs. Once converging to subject-specific values of δ* and N*, we then test the classifier solution for statistical significance using the false discovery rate across all windows , as there are approximately E/2δ* multiple comparisons for an E milliseconds epoch with 50% window overlap.
First, minimizing our objective function with N held constant at its maximum, we find that δ* can be tuned in a subject-specific way and we find on average a 3.7 ± 1.1% improvement in maximum Az from that of the earlier study. Second, we vary δ (δ ∈ [5, 6, ..., 149, 150]ms) and N (N ∈ [10, 11, ..., Nmax-1, Nmax] ) simultaneously and converge using a genetic algorithm (6-bit resolution, 36-member population, 0.7 crossover probability, 0.7/(population size) mutation probability, ) to a subject-specific δ* and N*. In each subject but one we find that N* < Nmax and that δ* is a subject-specific parameter that differs from the heuristics offered by previous work. Finally, on a group level, we find that the components of our objective function exhibit distinct variation with respect to δ and N, with an epoch's maximum Az optimizing for low N and low δ, while its Az variability minimizes for high N and maximizes for low N, nearly irrespective of δ.
- Philiastides MG, Sajda P: Temporal characterization of the neural correlates of perceptual decision making in the human brain. Cereb Cortex. 2006, 16 (4): 509-518.View ArticlePubMedGoogle Scholar
- Goldman RI, Wei CY, Philiastides MG, Gerson AD, Friedman D, Brown TR, Sajda P: Single-trial discrimination for integrating simultaneous EEG and fMRI: identifying cortical areas contributing to trial-to-trial variability in the auditory oddball task. NeuroImage. 2009, 47 (1): 136-147. 10.1016/j.neuroimage.2009.03.062.PubMed CentralView ArticlePubMedGoogle Scholar
- Sherwin J, Muraskin J, Sajda P: You can't think and hit at the same time: Neural correlates of baseball pitch classification. Frontiers of Neuroscience. 2012, 6 (177):Google Scholar
- Benjamini Y, Hochberg Y: Controlling the fasle discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B (Methodological). 1995, 57 (1): 289-300.Google Scholar
- Chipperfield J, Dakev NV, Fleming PJ, Whidborne JF: Multiobjective robust control using evolutionary algorithms. Proceedings of the Ieee International Conference on Industrial Technology (Icit'96). 1994, 269-273.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.