EEGManyPipelines - effekter av analytisk variabilitet på resultat i EEG-forskning
Electroencephalography (EEG) is widely used to investigate human cognition. However, the credibility of EEG findings has been called into question. Analysis pipelines are highly variable across studies, because of the many ways in which EEG data can be processed and analysed. The degree to which alternative, plausible pipelines yield different results and conclusions is currently unknown.
This proposal presents EEGManyPipelines, a large-scale international collaborative project addressing the robustness of EEG research by asking many independent teams to analyse the same data with an analysis pipeline they deem sensible and representative of their own research. Analysts will report their results and a detailed description of the analysis pipeline, allowing us to analyze the diversity of analysis pipelines and their effects on results. Thus, this project will help assess the robustness of EEG findings across alternative analyses, identifying (sub)optimal analysis pipelines, and informing guidelines for reporting EEG analyses in publications.
We expect to facilitate a cultural shift away from small-scale single laboratory experiments towards high-powered, community driven collaborations. This project will help improve the credibility of EEG findings and the quality of analyses, and will inspire new standards for conducting and reporting EEG studies, thereby supporting the foundation of future EEG research.
This proposal presents EEGManyPipelines, a large-scale international collaborative project addressing the robustness of EEG research by asking many independent teams to analyse the same data with an analysis pipeline they deem sensible and representative of their own research. Analysts will report their results and a detailed description of the analysis pipeline, allowing us to analyze the diversity of analysis pipelines and their effects on results. Thus, this project will help assess the robustness of EEG findings across alternative analyses, identifying (sub)optimal analysis pipelines, and informing guidelines for reporting EEG analyses in publications.
We expect to facilitate a cultural shift away from small-scale single laboratory experiments towards high-powered, community driven collaborations. This project will help improve the credibility of EEG findings and the quality of analyses, and will inspire new standards for conducting and reporting EEG studies, thereby supporting the foundation of future EEG research.
Final report
Project purpose and development
The EEGManyPipelines project was a large-scale effort to replicate experimental findings in EEG research by pooling data collection efforts across multiple laboratories, here, we rely on a multi-analyst approach to investigate EEG analysis practices: Many independent analysis teams test the same set of hypotheses on the same data and report their analyses in detail, providing a record of their results and analysis code. In that sense, our project targets EEG analyses as conducted “in the wild”: (1) It goes beyond summarizing analysis practices reported in published EEG studies by observing in detail how such analyses are conducted and implemented in actual research environments; (2) the analyses are executed by a large, representative sample of analysts rather than a single team; and (3) the analysts are granted the autonomy to make their own analytic choices, mirroring their own “natural” research work.
Execution
The project execution was managed by a steering committee including many early-career researchers. We recruited 168 analyst teams, encompassing 396 individual researchers from 37 countries, who independently analyzed the same unpublished, representative EEG data set to test the same set of predefined hypotheses and then provided their analysis pipelines and reported outcomes. The Steering Committee selected a dataset representative of a typical EEG experiment, comprising EEG data from 33 human participants who performed a long-term memory task. We formulated eight hypotheses for testing using this dataset, varying in their analysis modality and degree of specificity. We collected analysis code, preprocessed data, verbal descriptions of analyses, and analysts’ predictions of results.
Three most important results and discussion
No two submitted pipelines were identical in their preprocessing and analysis choices. The teams used a range of common EEG analysis packages and different reference schemes. Statistical analysis choices also varied considerably across teams.
Our findings revealed that (1) the existing variability across analysis pipelines reflects wide differences in researchers’ choices of analysis procedures and parameter settings; (2) nevertheless, researchers reached a high consensus (on average 76.9%) on whether event-related potential (ERP) hypotheses were confirmed, despite notable differences in effect sizes (p-values and difference waves); (3) certain preprocessing parameters, mainly the reference channel and high-pass filter cutoff, predicted difference wave magnitude; and (4) pipeline choices explained a limited amount of variance in p-values for two out of three hypotheses. Taken together, our results demonstrate variability in EEG analysis practice and consequently variation in reported effect sizes.
This large-scale, coordinated effort provides the first systematic quantification of how analytical choices shape EEG research outcomes under controlled conditions. This work establishes a new empirical foundation for understanding reproducibility in human neuroscience and underscores the need for transparent reporting, methodological standardization, and collaborative verification across research teams.
For the general fields of open science, meta-science, and meta-analyses, our findings suggest once again that projects that aim to aggregate singular findings into one overall effect estimate should focus on the actual (preprocessed) data used to make statistical decisions, rather than relying on binary significant/non-significant reports. This requires sharing of data, code, and appropriate documentation.
Any new research questions
We introduced a new prototypicality metric for use in future many analysts projects. We defined the prototypical pipeline in terms of step order and choices within each step separately, and quantified the deviation of each team’s pipeline from this prototype. Linking the new prototypicality metric to the observed difference wave, we observed that teams with more unusual pipelines (i.e., larger deviation from the prototypical step order) tended to observe weaker effects (i.e., smaller deviations of the difference waves from zero). Future research using simulated data in which the true underlying effect size is known is needed to determine whether an overall assessment of a pipeline as “typical” vs. “atypical” is a good first indicator for its appropriateness.
Dissemination
The project has been preregistered. We have published a position paper with initial data on the recruitment of analysts etc. Data will be openly shared on the BrainLife platform. A data descriptor manuscript and a main manuscript are under preparation. We have disseminated project information and results in numerous conferences, including the Society for the Improvement of Psychological Science (SIPS) and Psychologie und Gehirn (PuG) conferences.
Publications
Trübutschek, D., Yang, Y. F., Gianelli, C., Cesnaite, E., Fischer, N. L., Vinding, M. C., ... & Nilsonne, G. (2024). EEGManyPipelines: a large-scale, grassroots multi-analyst study of electroencephalography analysis practices in the wild. Journal of Cognitive Neuroscience, 36(2), 217-224. Published with open access.
https://eegmanypipelines.github.io/
The EEGManyPipelines project was a large-scale effort to replicate experimental findings in EEG research by pooling data collection efforts across multiple laboratories, here, we rely on a multi-analyst approach to investigate EEG analysis practices: Many independent analysis teams test the same set of hypotheses on the same data and report their analyses in detail, providing a record of their results and analysis code. In that sense, our project targets EEG analyses as conducted “in the wild”: (1) It goes beyond summarizing analysis practices reported in published EEG studies by observing in detail how such analyses are conducted and implemented in actual research environments; (2) the analyses are executed by a large, representative sample of analysts rather than a single team; and (3) the analysts are granted the autonomy to make their own analytic choices, mirroring their own “natural” research work.
Execution
The project execution was managed by a steering committee including many early-career researchers. We recruited 168 analyst teams, encompassing 396 individual researchers from 37 countries, who independently analyzed the same unpublished, representative EEG data set to test the same set of predefined hypotheses and then provided their analysis pipelines and reported outcomes. The Steering Committee selected a dataset representative of a typical EEG experiment, comprising EEG data from 33 human participants who performed a long-term memory task. We formulated eight hypotheses for testing using this dataset, varying in their analysis modality and degree of specificity. We collected analysis code, preprocessed data, verbal descriptions of analyses, and analysts’ predictions of results.
Three most important results and discussion
No two submitted pipelines were identical in their preprocessing and analysis choices. The teams used a range of common EEG analysis packages and different reference schemes. Statistical analysis choices also varied considerably across teams.
Our findings revealed that (1) the existing variability across analysis pipelines reflects wide differences in researchers’ choices of analysis procedures and parameter settings; (2) nevertheless, researchers reached a high consensus (on average 76.9%) on whether event-related potential (ERP) hypotheses were confirmed, despite notable differences in effect sizes (p-values and difference waves); (3) certain preprocessing parameters, mainly the reference channel and high-pass filter cutoff, predicted difference wave magnitude; and (4) pipeline choices explained a limited amount of variance in p-values for two out of three hypotheses. Taken together, our results demonstrate variability in EEG analysis practice and consequently variation in reported effect sizes.
This large-scale, coordinated effort provides the first systematic quantification of how analytical choices shape EEG research outcomes under controlled conditions. This work establishes a new empirical foundation for understanding reproducibility in human neuroscience and underscores the need for transparent reporting, methodological standardization, and collaborative verification across research teams.
For the general fields of open science, meta-science, and meta-analyses, our findings suggest once again that projects that aim to aggregate singular findings into one overall effect estimate should focus on the actual (preprocessed) data used to make statistical decisions, rather than relying on binary significant/non-significant reports. This requires sharing of data, code, and appropriate documentation.
Any new research questions
We introduced a new prototypicality metric for use in future many analysts projects. We defined the prototypical pipeline in terms of step order and choices within each step separately, and quantified the deviation of each team’s pipeline from this prototype. Linking the new prototypicality metric to the observed difference wave, we observed that teams with more unusual pipelines (i.e., larger deviation from the prototypical step order) tended to observe weaker effects (i.e., smaller deviations of the difference waves from zero). Future research using simulated data in which the true underlying effect size is known is needed to determine whether an overall assessment of a pipeline as “typical” vs. “atypical” is a good first indicator for its appropriateness.
Dissemination
The project has been preregistered. We have published a position paper with initial data on the recruitment of analysts etc. Data will be openly shared on the BrainLife platform. A data descriptor manuscript and a main manuscript are under preparation. We have disseminated project information and results in numerous conferences, including the Society for the Improvement of Psychological Science (SIPS) and Psychologie und Gehirn (PuG) conferences.
Publications
Trübutschek, D., Yang, Y. F., Gianelli, C., Cesnaite, E., Fischer, N. L., Vinding, M. C., ... & Nilsonne, G. (2024). EEGManyPipelines: a large-scale, grassroots multi-analyst study of electroencephalography analysis practices in the wild. Journal of Cognitive Neuroscience, 36(2), 217-224. Published with open access.
https://eegmanypipelines.github.io/