1. PROJECT ------------ Title: The Psychology of Philosophical Thought Experiments Dates: 01/04/2017 – 30/04/2019 Funding organisation: The Leverhulme Trust Grant no.: RPG-2016-193 2. DATASET ------------ Title: Stakes, Scales, and Skepticism Datasets (The Psychology of Philosophical Thought Experiments) Description: The datasets included here were collected online as part of the project: “The psychology of philosophical thought experiments” and are forthcoming in "Stakes, Scales, and Skepticism" in Ergo, 2019. In this project, we were interested in investigating whether there are “stakes effects on knowledge” – which posits that the “stakes” or importance of being wrong affects judgments about whether a subject knows a proposition. Publication Year: 2019 Creator(s): Kathryn B. Francis, Nat Hansen, Philip Beaman Organisation(s): University of Reading Rights-holder(s): Kathryn B. Francis, Nat Hansen, Philip Beaman 3. TERMS OF USE ----------------- Copyright 2019 Kathryn B. Francis, Nat Hansen, Philip Beaman. This dataset is licensed by the rights-holders under a Creative Commons Attribution 4.0 International Licence: https://creativecommons.org/licenses/by/4.0/. 4. CONTENTS ------------ We include specific descriptions of the datasets below according to the folders in which they are located. The datasets contain full (and adjusted datasets) for each of the experiments described and referenced in: Francis, K.B., Beaman, C.P., & Hansen, N. Stakes, Scales, and Skepticism (Ergo, 2019). Each sfolder containing the data from a specific experiment is labelled using the experiment name given in the paper: ‘Exp1_Evidence_Fixed_Design.zip’ ‘Exp2_Evidence_Seeking_Design.zip’ ‘Exp2.1_Symmetrical_Experiment.zip’ ‘Exp2.2_Matched_Experiment.zip’ ‘Replication_Experiment.zip’ Datasets are preserved in .CSV format (comma delimited) and the first row of the datasets contains clear column headings. For each experiment, there are three .CSV files saved: 1) All original data collected from Amazon Mechanical Turk These .CSV files contain all data exported from Qualtrics/MTurk prior to suspicious response checks and control prompt checks. These files will have the phrase ‘pre’ in their name. 2) Final data for the ‘positive’ polarity condition These .CSV files contain the final dataset in the positive polarity condition (‘know’). These files will have the phrase ‘Pos’ in their name. 3) Final data for the ‘negative’ polarity condition These .CSV files contain the final dataset in the negative polarity condition (‘don’t know’). These files will have the phrase ‘Neg’ in their name. The 2) and 3) datasets above have been adjusted in the following ways: a. Individuals are removed from the dataset if their responses are flagged as being suspicious(‘suspicious’ = responses suspected to derive from a server farm or duplicate responses). b. Individuals are removed from the dataset if they failed the first and/or second control prompt check. The ‘Replication Experiment’ folder contains two .CSV files (one containing original data ('pre') and one with data removed following MTurk suspicious response checks ('post') (described above)). File listing: Experimental datasets from paper: Francis, K.B., Beaman, C.P., & Hansen, N. Stakes, Scales, and Skepticism (Ergo, 2019). Folder: ‘Exp1_Evidence_Fixed_Design.zip’ File: x1_EF_data_pre_removal.CSV File: x1_EF_Neg_data_post_removal.CSV File: x1_EF_Pos_data_post_removal.CSV Folder: ‘Exp2_Evidence_Seeking_Design.zip’ File: x2_ES_data_ pre_removal.CSV File: x2_ES_Neg_data_post_removal.CSV File: x2_ES_Pos_data_post_removal.CSV Folder: ‘Exp2.1_Symmetrical_Experiment.zip’ File: x2.1_Sym_data_pre_removal.CSV File: x2.1_Sym_Neg_data_post_removal.CSV File: x2.1_Sym_Pos_data_post_removal.CSV Folder: ‘Exp2.2_Matched_Experiment.zip’ File: x2.2_Matc_data_pre_removal.CSV File: x2.2_Matc_Neg_data_post_removal.CSV File: x2.2_Matc_Pos_data_post_removal.CSV Folder: ‘Replication_Experiment.zip’ File: xReplic_data_pre_removal.CSV File: xReplic_data_post_removal.CSV 5. METHOD and PROCESSING -------------------------- ‘Stakes, Scales, and Skepticism’ All data stored was collected online: -Qualtrics was used to present scenarios and questions -Amazon Mechanical Turk was used to recruit participants and distribute the Qualtrics experiment. The experimental procedure used in Qualtrics can be found in ‘Appendix I’. Data were altered/processed in the following way: 1)Across all experimental datasets, individuals are removed if they failed to successfully respond to one or both control prompts: Control prompt 1: In this experiment, you will read several scenarios and then be asked to respond to different statements about those scenarios. So, for example, consider the following scenario: You have a fair coin, with heads on one side and tails on the other, that you flip into the air and catch on the back of your hand without looking at it. Please indicate whether you agree or disagree with the following statement about the scenario: You know the coin landed heads [1-strongly agree – 7-strongly disagree] Failure = 1 or 2. Control prompt 2: In this experiment, you will read several scenarios and then be asked to respond to different statements about those scenarios. So, for example, consider the following scenario: You have a fair coin, with heads on one side and tails on the other, that you flip into the air and catch on the back of your hand without looking at it. Please indicate whether you agree or disagree with the following statement about the scenario: You don’t know that the coin landed heads [1-strongly agree – 7-strongly disagree] Failure = 6 or 7. 2)‘Suspicious’ responses from MTurk are flagged using the following procedure: 1. Identifying identical GPS locations with unique IP addresses 2. Using IP address locator to determine whether these IP addresses derive from an Internet Service Provider (ISP) or data center (VPS) 3. Open-ended responses are then evaluated against a set of criteria (full details regarding this procedure: Dennis, Goodson, & Pearson (August 17, 2018) Note that for Experiments 2, 2.1, and 2.2, analyses are reported in the paper following outlier removal (analyses prior to outlier removal are included in an appendix). This removal was performed using SPSS and R. Datasets were log-transformed and outlier detection performed on the normalised distributions. Outliers were identified as those outside the range of: median +/- 2.5 x Median Absolute Deviation (MAD). Analyses were then performed on the non-transformed skewed data with outliers now removed. 6. APPENDIX ------------ Experimental Flow (Exp 1, Exp 2, Exp 2.1, Exp 2.2): Blocks Introduction: Thank you for your interest in this short survey. Participation involves reading several short stories and answering a question about each, and then providing basic demographic information. It should take 10-20 minutes to complete. There is no risk to the participant. You must be at least 18 years old to participate. The survey is only open to participants whose first language is English. You may withdraw from the study at any time by closing your web browser. All data provided are confidential. The results of the study may be published in an academic journal. This project has been subject to ethical review, according to the procedures specified by the University Research Ethics Committee, and has been given a favorable ethical opinion for conduct. Consent: Please answer the following to consent to take part in the study I have read the study information, understand the purposes of the project and what will be required of me. I agree to the arrangements described in the study information in so far as they relate to my participation. [YES/NO] I understand that participation is entirely voluntary and that I have the right to withdraw from the project any time, and that this will be without detriment. [YES/NO] I understand that this project has been subject to ethical review, according to the procedures specified by the University Research Ethics Committee, and has been given a favorable ethical opinion for conduct. [YES/NO] Block 12: Control Prompt Check 1 Block 13: Ok great! On the following pages, please read the scenarios and respond to the statements about them as instructed. Scenarios: Block A = “Know” Block A2 = “Don’t Know” Paramedic________Low Stakes Vaccine__________Stakes Two Mountaineering___Stakes Three Game Show________High Stakes Introductions____Low Stakes Possessions______Stakes One Block B = “Know” Block B2 = “Don’t Know” Paramedic________Stakes Two Vaccine__________Stakes Three Mountaineering___High Stakes Game Show________Low Stakes Introductions____Stakes Two Possessions______Stakes Three Block C = “Know” Block C2 = “Don’t Know” Paramedic________Stakes Three Vaccine__________High Stakes Mountaineering___Low Stakes Game Show________Stakes Two Introductions____Stakes Three Possessions______High Stakes Block D = “Know” Block D2 = “Don’t Know” Paramedic________High Stakes Vaccine__________Low Stakes Mountaineering___Stakes Two Game Show________Stakes Three Introductions____High Stakes Possessions______Low Stakes Block 15: “Great! You’re almost done” Block 14: Control Prompt Check 1 Demographics: What do you think the purpose of this study was? [text response] What is your native language [English, Other (please specify)] Your gender (Male, Female, Other (e.g., non-binary)] What is your age? [text response] Have you ever studied philosophy at university level? [YES/NO] MTurk ID: Enter your MTurk Worker ID [text response] Thank you for taking part in this survey. In order to complete the survey and receive your credit through MTurk please advance to the next page of the survey which will provide you with a random number. Experimental Flow (Replication Experiment): Blocks Introduction: Thank you for your interest in this short survey. Participation involves reading 1 short scenario and answering 2 questions about it, and then providing basic demographic information. It should take 5 minutes to complete. There is no risk to the participant. You must be at least 18 years old to participate. The survey is only open to participants whose first language is English. You may withdraw from the study at any time by closing your web browser. All data provided are confidential. The results of the study may be published in an academic journal. This project has been subject to ethical review, according to the procedures specified by the University Research Ethics Committee, and has been given a favorable ethical opinion for conduct. Consent: Please answer the following to consent to take part in the study I have read the study information, understand the purposes of the project and what will be required of me. I agree to the arrangements described in the study information in so far as they relate to my participation. [YES/NO] I understand that participation is entirely voluntary and that I have the right to withdraw from the project any time, and that this will be without detriment. [YES/NO] I understand that this project has been subject to ethical review, according to the procedures specified by the University Research Ethics Committee, and has been given a favorable ethical opinion for conduct. [YES/NO] Ready Block: Ok great! On the following page, please read the scenario and respond to the statements about it as instructed. Basic Low Stakes: As described in: Sripada, S. and Stanley, J. (2012). “Empirical Tests of Interest-Relative Invariantism”, Episteme, 9(1), 3-26. Basic High Stakes: As described in: Sripada, S. and Stanley, J. (2012). “Empirical Tests of Interest-Relative Invariantism”, Episteme, 9(1), 3-26. Implicit Low Stakes: As described in: Sripada, S. and Stanley, J. (2012). “Empirical Tests of Interest-Relative Invariantism”, Episteme, 9(1), 3-26. Explicit High Stakes: As described in: Sripada, S. and Stanley, J. (2012). “Empirical Tests of Interest-Relative Invariantism”, Episteme, 9(1), 3-26. Ignorant Low Stakes: As described in: Sripada, S. and Stanley, J. (2012). “Empirical Tests of Interest-Relative Invariantism”, Episteme, 9(1), 3-26. Ignorant High Stakes: As described in: Sripada, S. and Stanley, J. (2012). “Empirical Tests of Interest-Relative Invariantism”, Episteme, 9(1), 3-26. Demographics: What do you think the purpose of this study was? [text response] What is your native language [English, Other (please specify)] Your gender (Male, Female, Other (e.g., non-binary)] What is your age? [text response] Have you ever studied philosophy at university level? [YES/NO] MTurk ID: Enter your MTurk Worker ID [text response] Thank you for taking part in this survey. In order to complete the survey and receive your credit through MTurk please advance to the next page of the survey which will provide you with a random number.