Resumen de Statistical analysis and planning of brain decoding studies: insights from developing an attentional brain-computer interface
Filip Melinscak
Reproducibility is generally considered to be a cornerstone of the scientific method. However, in the recent years, concerns over reproducibility have emerged across a number of different fields. Reproducibility of scientific findings was first questioned by methodologists, who have voiced concerns about the perilous combination of publication bias, over-reliance on null hypothesis significance testing (NHST, i.e. p-values), and low statistical power. An increasing number of failed replications across diverse scientific fields -- psychology, medicine, experimental economics -- has soon justified the worries about common scientific methodologies, leading to the so-called reproducibility crisis. Importantly, issues with reproducibility and statistical methods have also been reported in the areas of neuroscience and neuroimaging. Motivated by these methodological issues, this thesis rethinks the statistical methods employed in a sub-field of neuroimaging -- brain decoding.
Brain decoding is the application of machine learning techniques, in the form of decoding models, to the problem of predicting or detecting aspects of experiential context from recorded neural data. Brain decoding approaches are increasingly popular with a number of different neuroimaging modalities, such as fMRI and EEG.
Most research efforts in the development of brain decoding models is usually devoted to the technical implementation of such models. However, as the reproducibility crisis has demonstrated, the planning stage of the experiment (e.g. sample size planning), and the appropriate statistical analysis of the experimental results can be crucial for the validity of findings. And yet, these statistical aspects of developing brain decoding models are often neglected in practice. Hence, the main objective of this thesis was to develop a comprehensive statistical framework for the analysis and planning of brain decoding studies.
The practical motivation to focus our efforts on the statistical aspects of brain decoding came from a concrete engineering challenge: developing a passive brain-computer interface (BCI) for the estimation of attentional states. Brain-computer interfaces can be considered to be a particular application-oriented area of brain decoding. One of the emerging clinical applications of BCI technology is rehabilitation after brain injuries, such as stroke. In the context of this trend, the auxiliary goal of this thesis was to develop an EEG-based BCI capable of discriminating between kinesthetic attention and mind wandering. Such a BCI could be used in order to make rehabilitation therapy more adaptive to the attentional state of the patient.
In order to develop an attentional BCI for rehabilitation applications, we have first designed a novel protocol emulating a rehabilitation scenario with the passive mobilization of lower limbs. The protocol was self-paced and featured alternating periods of sustained attention and deliberate mind wandering, thus being more ecologically valid than typical trial-based protocols. The EEG data was collected from ten subjects, and the analysis of electrophysiology showed that the spectral power in the alpha and beta frequency bands was lower in the attentive state, compared to the state of deliberate mind wandering. Based on these spectral features, an asynchronous BCI was implemented and validated in simulated online conditions. The obtained accuracies varied between 60% and 70% at the group level, depending on the settings of the BCI.
In the course of developing the aforementioned attentional BCI, we have identified some troubling aspects with usual methods for statistical analysis of BCI performances, based on NHST and p-values. The thesis therefore rethinks the value of NHST, both in the context of the reproducibility crisis, and BCI research practice. As an alternative to NHST, we propose Bayesian methods and motivate them starting from first principles of statistical inference. In particular, we propose Bayesian estimation as the appropriate framework for the analysis of BCI performance data. The framework of Bayesian estimation is then used to develop models of BCI performance for three most common experimental designs in the field. Moreover, the three models are shown to be special cases of the hierarchical generalized linear model (HGLM) which is proposed as a flexible statistical model which can be extended to a variety of BCI experimental designs. We have then proceeded to validate the framework of Bayesian estimation by reanalyzing three publicly available BCI performance datasets and by revisiting the results of the attentional BCI. The framework of Bayesian estimation provided results which were arguably more interpretable and informative than results of NHST on the same datasets.
As mentioned before, another statistical factor that is generally thought to be related to the reproducibility crisis is the issue of adequate sample sizes and experimental designs, both of which influence statistical power. Although tools for sample size planning have been proposed in the area of neuroimaging, methods for design optimization that would be tailored towards brain decoding studies have not yet been developed. Hence, in this thesis we propose a simulation-based method of optimizing experimental designs and we use it for sample size determination of brain decoding and BCI studies. The method was validated through extensive simulations. The simulations showed that optimal experimental designs crucially depend on the available prior knowledge and on the goals of the experiments, thus justifying the use of formal optimization methods instead of ad-hoc rules of thumb. The simulations also showed that optimized designs have the potential to be substantially more efficient than sub-optimal designs. As an additional validation of the method, we used it to plan a hypothetical replication of the study performed during the development of the attentional BCI. By incorporating the information obtained from the available BCI performance data, we show that the replication study would have a rather different and more efficient sampling plan.
The proposed methodology of using Bayesian estimation and simulation-based experimental design forms a comprehensive statistical framework for the analysis and planning of brain decoding and BCI studies, thus fulfilling the main goal of this thesis. The methodology inherits the attractive theoretical properties of Bayesian statistics and brings a number of practical advantages, such as flexibility, interpretability, and the accumulation of knowledge across studies. Although the proposed framework cannot be used to completely solve the issues of reproducibility, we consider it to be competitive with the classic NHST tools used in BCI and brain decoding research. Moreover, the proposed framework is general enough that it may potentially be useful in other scientific fields which use machine learning techniques and evaluate them through empirical experimentation.
