Questionable Research Practices: The (un)ethical handling of data in quantitative humanities research
Questionable Research Practices (QRPs) are often viewed as the “murky waters” of research ethics. On the scale from “ideal behavior” to absolute misconduct, QRPs make up the grey area in-between. As such, they can be difficult to detect, and, what is worse, researchers may not even be aware of the fact that they are doing something wrong. For example, with increasing pressure to secure external funding, choosing to tone down findings that go against the interests of the grant-funding agency may seem like the best solution. However, misrepresenting information or failing to objectively report it are examples of QRPs and should thus be avoided.
The present project aims to bring clarity to the concept of QRPs as it relates to the humanities, thereby providing domain-specific tools to help Swedish researchers avoid such practices. The project also includes a systematic comparison with one of the main actors on the research scene: the U.S. Specifically, the project (i) investigates the range and severity of QRPs faced by researchers in the field, (ii) surveys researchers’ experiences with QRPs, and (iii) assesses the extent to which training for PhD students addresses these QRPs. Based on the results, four articles will be published and humanities-specific training materials in QRPs will be developed. At a general level, the project will help raise researchers’ awareness of QRPs, thus enabling high-quality research to be performed in a more ethical as well as informative manner.
Final report
Questionable research practices: The (un)ethical handling of data in quantitative humanities research (FOE20-0017).
Final report
This project was designed to investigate the prevalence, frequency, and perceived severity of so-called Questionable Research Practices (QRPs) in quantitative humanities research. QRPs can be found in the gray zone between responsible research conduct, on the one hand, and falsification, fabrication, and plagiarism, on the other hand.
The project was implemented through three phrases. Phase 1 involved the development of a taxonomy of QRPs, with the goal of obtaining a comprehensive list of items relevant to quantitative humanities research. To do so, we employed the Delphi method, a method that has not been used much in our field. The Delphi method can be thought of as an asynchronous focus group where a team of experts is invited to weigh in on which items should be added to the taxonomy and how they should be worded. Somewhat simplified, the experts (10 researchers from relevant fields who have expertise in research ethics) proposed items that we (the research team) summarized; we also complemented the list with items from existing taxonomies from other fields. The experts then provided feedback on the wording of the items during multiple rounds of review. The resulting taxonomy encompasses 58 items, 90 percent of which came from the experts. The items are grouped into four categories: funding, design/data collection, data analysis, and dissemination. During this phase, we also carried out a thematic overview of research ethics in our field.
In Phase 2, we used the newly developed taxonomy to survey the broad field of quantitative humanities, with the goal of better understanding the prevalence, frequency, and perceived severity of these QRPs among researchers. The survey was sent out to researchers with a PhD who are currently active at ‘research-intensive’ institutions in Sweden or the US. For Sweden, we sent the survey to all universities (universitet) that have faculty doing research in the quantitative humanities; for the US, we sent it to research 1 or research 2 (R1 or R2) institutions. Our sample (n = 230) covered disciplines such as applied linguistics, education, history, gender studies, media studies, and philosophy.
Phase 3 involved developing teaching materials based on our findings from Phases 1 and 2. We recorded instructional videos introducing each QRP, and produced sample lesson plans and an activity bank with classroom activities designed to help teachers implement ethics training into their classes and curricula. During this phase, we also carried out an overview of existing teaching materials in the field by looking at course syllabi and textbooks.
The most important findings were as follows:
•Although research ethics has been a topic that has been discussed in the field since the 1980s, it was not until the 2010s that we started seeing more wide-spread interest in researching the topic.
•QRPs are highly prevalent: 96 percent of the participants reported having done at least one of the QRP items at least once in the past five years.
•The QRPs from the taxonomy that were most frequent were as follows:
-Presenting the same presentation at multiple conferences
-Defaulting to convention (e.g., choosing a design or instrument type because it is used in previous research, without making sure that it is the most appropriate design or instrument for the target relationships and/or constructs)
•The QRPs from the taxonomy that were least frequent were as follows:
-Not reporting a conflict of interest (financial or otherwise)
-Using unjustified methods of handling missing data
•The QRPs from the taxonomy that were considered least severe were as follows:
-Presenting the same presentation at multiple conferences
-Defaulting to convention
•The QRPs from the taxonomy that were considered most severe were as follows:
-Not reporting a conflict of interest
-Removing whole items/cases knowingly/purposefully to obtain favorable results
•There was a strong negative correlation between frequency and perceived severity of the QRPs from the taxonomy (Spearman’s rho = -0.77).
•Existing coverage of research ethics in courses and textbooks tended to focus primarily on (i) research ethics as exclusively ethics-board focused, (ii) methodological decisions (e.g., selection of appropriate statistical tests and data sharing and open science practices), and (iii) research ethics as a specialized topic (i.e., covered in just one chapter/class, rather than as a topic of relevance to all stages of the research process).
We thus saw that research ethics as a subfield within quantitative humanities research is gaining momentum. We also found that research ethics tends to be treated in a compartmentalized manner, with existing training materials mostly focusing on isolated topics in a single chapter or class. For QRPs specifically, we noted that they are prevalent in the field. There was a strong negative correlation between frequency and perceived severity, suggesting that researchers do not do QRPs that they consider to be severe. Based on the findings, we propose that more formal training would be beneficial for the field, and we suggest that this training integrate discussions of research ethics (and QRPs) into all stages of the research process. Our goal with the training materials that we produced is to encourage reflection about QRPs, not provide clear-cut answers, as such answers rarely exist in the realm of QRPs.
Our research has been disseminated in several different ways: we have published one book of training materials, five peer-reviewed articles in prestigious journals, and we have given ten conference presentations. We have also organized three workshops with invited speakers and audience members from a wide range of departments: two at Uppsala University in Sweden and one at Northern Arizona University in the US. Discussions carried out during these events resulted in a collaborative publication where we teamed up with presenters from these events. In addition, we have a social media account on X where we have posted updates about the project and a website where we post updates about this project along with related, more recent projects.
Final report
This project was designed to investigate the prevalence, frequency, and perceived severity of so-called Questionable Research Practices (QRPs) in quantitative humanities research. QRPs can be found in the gray zone between responsible research conduct, on the one hand, and falsification, fabrication, and plagiarism, on the other hand.
The project was implemented through three phrases. Phase 1 involved the development of a taxonomy of QRPs, with the goal of obtaining a comprehensive list of items relevant to quantitative humanities research. To do so, we employed the Delphi method, a method that has not been used much in our field. The Delphi method can be thought of as an asynchronous focus group where a team of experts is invited to weigh in on which items should be added to the taxonomy and how they should be worded. Somewhat simplified, the experts (10 researchers from relevant fields who have expertise in research ethics) proposed items that we (the research team) summarized; we also complemented the list with items from existing taxonomies from other fields. The experts then provided feedback on the wording of the items during multiple rounds of review. The resulting taxonomy encompasses 58 items, 90 percent of which came from the experts. The items are grouped into four categories: funding, design/data collection, data analysis, and dissemination. During this phase, we also carried out a thematic overview of research ethics in our field.
In Phase 2, we used the newly developed taxonomy to survey the broad field of quantitative humanities, with the goal of better understanding the prevalence, frequency, and perceived severity of these QRPs among researchers. The survey was sent out to researchers with a PhD who are currently active at ‘research-intensive’ institutions in Sweden or the US. For Sweden, we sent the survey to all universities (universitet) that have faculty doing research in the quantitative humanities; for the US, we sent it to research 1 or research 2 (R1 or R2) institutions. Our sample (n = 230) covered disciplines such as applied linguistics, education, history, gender studies, media studies, and philosophy.
Phase 3 involved developing teaching materials based on our findings from Phases 1 and 2. We recorded instructional videos introducing each QRP, and produced sample lesson plans and an activity bank with classroom activities designed to help teachers implement ethics training into their classes and curricula. During this phase, we also carried out an overview of existing teaching materials in the field by looking at course syllabi and textbooks.
The most important findings were as follows:
•Although research ethics has been a topic that has been discussed in the field since the 1980s, it was not until the 2010s that we started seeing more wide-spread interest in researching the topic.
•QRPs are highly prevalent: 96 percent of the participants reported having done at least one of the QRP items at least once in the past five years.
•The QRPs from the taxonomy that were most frequent were as follows:
-Presenting the same presentation at multiple conferences
-Defaulting to convention (e.g., choosing a design or instrument type because it is used in previous research, without making sure that it is the most appropriate design or instrument for the target relationships and/or constructs)
•The QRPs from the taxonomy that were least frequent were as follows:
-Not reporting a conflict of interest (financial or otherwise)
-Using unjustified methods of handling missing data
•The QRPs from the taxonomy that were considered least severe were as follows:
-Presenting the same presentation at multiple conferences
-Defaulting to convention
•The QRPs from the taxonomy that were considered most severe were as follows:
-Not reporting a conflict of interest
-Removing whole items/cases knowingly/purposefully to obtain favorable results
•There was a strong negative correlation between frequency and perceived severity of the QRPs from the taxonomy (Spearman’s rho = -0.77).
•Existing coverage of research ethics in courses and textbooks tended to focus primarily on (i) research ethics as exclusively ethics-board focused, (ii) methodological decisions (e.g., selection of appropriate statistical tests and data sharing and open science practices), and (iii) research ethics as a specialized topic (i.e., covered in just one chapter/class, rather than as a topic of relevance to all stages of the research process).
We thus saw that research ethics as a subfield within quantitative humanities research is gaining momentum. We also found that research ethics tends to be treated in a compartmentalized manner, with existing training materials mostly focusing on isolated topics in a single chapter or class. For QRPs specifically, we noted that they are prevalent in the field. There was a strong negative correlation between frequency and perceived severity, suggesting that researchers do not do QRPs that they consider to be severe. Based on the findings, we propose that more formal training would be beneficial for the field, and we suggest that this training integrate discussions of research ethics (and QRPs) into all stages of the research process. Our goal with the training materials that we produced is to encourage reflection about QRPs, not provide clear-cut answers, as such answers rarely exist in the realm of QRPs.
Our research has been disseminated in several different ways: we have published one book of training materials, five peer-reviewed articles in prestigious journals, and we have given ten conference presentations. We have also organized three workshops with invited speakers and audience members from a wide range of departments: two at Uppsala University in Sweden and one at Northern Arizona University in the US. Discussions carried out during these events resulted in a collaborative publication where we teamed up with presenters from these events. In addition, we have a social media account on X where we have posted updates about the project and a website where we post updates about this project along with related, more recent projects.