# 1. ABOUT THE DATASET ------------ Title: Inter-rater reliability of functional MRI data quality control assessments dataset Creator(s): Brendan Williams [1,2], Nicholas Hedger [1,2], Carolyn B McNabb [3], Gabriella MK Rossetti [1,2]. Organisation(s): 1. School of Psychology and Clinical Language Sciences, University of Reading. 2. Centre for Integrative Neuroscience and Neurodynamics, University of Reading. 3. Cardiff University Brain Research Imaging Centre, Cardiff University. Rights-holder(s): University of Reading, Cardiff University. Publication Year: 2023 Description: This dataset contains rating information from four raters who reviewed the quality of functional magnetic resonance imaging (fMRI) datasets. These data were task-based and resting state fMRI data that were included in part of the fMRI Open QC Project [1], which aimed to showcase examples of QC practices across institutions and to foster discussions within the field. Raters made quality control decisions based on output generated by the python package, pyfMRIqc [2], which generates user friendly reports to aid decision making on the quality of fMRI data. Raters classified data using one of the three following criteria: Include, Uncertain, Exclude, based on guidance provided by a protocol included in the publication. Raters also provided notes describing their decision making for uncertain or exclude cases. A summary of the majority decision of all four raters is also included in this dataset for each subject. The reports generated by pyfMRIqc can be found externally on GitHub [3]. Cite as: Williams, Brendan., Hedger, Nicholas., McNabb, Carolyn B., and Rossetti, Gabriella MK. (2023): Inter-rater reliability of functional MRI data quality control assessments dataset. University of Reading. Dataset. https://doi.org/10.17864/1947.000424. Related publication: Williams, Brendan., Hedger, Nicholas., McNabb, Carolyn B., Rossetti, Gabriella MK., Christakou, Anastasia. (2023): Inter-rater reliability of functional MRI data quality control assessments: A standardised protocol and practical guide using pyfMRIqc. Frontiers in Neuroscience (In Press). Contact: b.williams3@reading.ac.uk # 2. TERMS OF USE ----------------- Copyright 2023 University of Reading and Cardiff University. All data and documentation are licensed by the rights-holder under a Creative Commons Attribution 4.0 International (CC BY 4.0) license (https://creativecommons.org/licenses/by/4.0/). # 3. PROJECT AND FUNDING INFORMATION ------------ Title: fMRI Open QC Project: Inter-rater reliability of functional MRI data quality control assessments Funding organisation: University of Reading. # 4. CONTENTS ------------ ## qc_decision_rater[1-4].csv: These four csv files contain quality control decision information for each of the four raters. Each csv file contains the same five columns: 'Subject', 'Include', 'Uncertain', 'Exclude', Notes. The 'Subject' column gives the subject ID from the fMRI Open QC Project dataset that each row corresponds to. The 'Include', 'Uncertain', and 'Exclude' columns are used record the QC classification of the rater for a given subject. A '1' is given in the column of the classification asigned by the rater, the other two columns are left empty. The 'Notes' column contains any notes made by the rater during the assessment of that dataset. ## rater_assignments.xlsx: Raters were assigned to review QC reports for 104 of the 129 subjects in the included dataset. This excel spreadsheet describes the assignment of subjects for each rater, denoted by a 1. Subject assignment ensured at least four subjects from each site were reviewed by all four raters, and every other subject was reviewed by three raters. Assignments were also balanced so that the proportion of overlapping cases was equal across raters. ## group_majority_qc_decision.csv: This csv file details the majority decision made by raters for the classification of subjects. There are two columns included in this csv file, 'Subject', which gives the subject ID from the fMRI Open QC Project dataset, and 'Majority_qc_decision', which details the majority decision among raters. The value 'I' is used for a majority include decision, 'U' for uncertain, and 'E' for exclude. # 5. METHODS -------------------------- Quality control assessments were completed by four independent raters (BW, NH, CBM GMKR), who were all postdoctoral research fellows, and all raters had previous experience in quality assessment, processing and analysis of functional neuroimaging data. Two raters (BW & GMKR) had previously used pyfMRIqc to perform quality assessment of fMRI data. Additionally, BW was involved in the development of pyfMRIqc. Each rater reviewed data for 104 of the 129 subjects, using outputs from cinnqc and pyfMRIqc. Raters were given the following instructions before beginning quality assessment: The following criteria need to be used to classify all images: Include - no quality assessment issues that indicate the dataset is problematic. Uncertain - some quality assessment issues that makes the inclusion of dataset marginal. Exclude - quality assessment issues that mean the data should not be included. Each image classified as either 'uncertain' or 'exclude' should include an explanation of why the given classification was made. Please be as descriptive as possible when explaining your decision-making. Quality assessment decision-making should be supported by the output produced by cinnqc and pyfMRIqc. cinnqc and pyfMRIqc derivatives can be found online in the directories /cinnqc/examples/{fmriqc-open-qc-task, fmriqc-open-qc-rest-100, fmriqc-open-qc-rest-200, fmriqc-open-qc-rest-300, fmriqc-open-qc-rest-400, fmriqc-open-qc-rest-500, fmriqc-open-qc-rest-600, fmriqc-open-qc-rest-700}/derivatives/cinnqc/ of the cinnqc GitHub page (https://github.com/bwilliams96/cinnqc).