This article developed from an exchange on Twitter (X), when one of us vented frustration about reviewers “not getting” qualitative research, and requesting revisions incongruent with the methodological approach of our submitted research. The frustration was not ours alone! After an animated exchange, we decided a more rigorous exploration of incongruity in peer review of qualitative research was needed, and so began a project to understand what appeared to be a widely shared problem. Most research on authors’ experiences of peer review and poor peer review practices has focused on science, technology, engineering, and mathematics disciplines (Atjonen, 2019) or implicitly invokes a model of peer review compatible only with hard sciences/quantitative research (e.g., Allen et al., 2019), with little attention to qualitative researchers’ experiences of peer review and poor peer review practices.

Peer Review (of Qualitative Research)—Entrenched but Broken?

Peer review is a cornerstone of academic publication, understood as both assuring quality and bolstering the trustworthiness of our knowledge production (Tennant & Ross-Hellauer, 2020). However, it is already widely acknowledged that peer review is flawed. Not only is there the challenge of too many articles and too few experts to review them (Rodríguez-Bravo et al., 2017), the process is resource intensive (Aczel et al., 2021), lacking transparency (O’Brien et al., 2021), inequitable (Steinberg et al., 2018), and at times actively hostile (Mavrogenis et al., 2020). Research on authors’ experiences of peer review has highlighted the harms of peer review for authors and particularly early career academics (e.g., Horn, 2016; Watling et al., 2023), and those from marginalized groups (e.g., Silbiger & Stubler, 2019). Poor peer review practices evidenced include substandard, contradictory, and destructive feedback (Atjonen, 2018) and feedback based on the personal biases or different schools of thought of the reviewers and editors (McCloskey & Merz, 2022; Overall, 2015). Authors perceive poor peer review practices as being at least partly explained by badly chosen reviewers (Jamali et al., 2020) and deficiencies in reviewer competence, including a lack of familiarity with the authors’ methods and paradigms (Atjonen, 2018, 2019; Watling et al., 2023). Problematic peer review can lead authors to make changes they may disagree with and may also consider as inaccurate, in order to get their article published (McCloskey & Merz, 2022; Overall, 2015). Describing the system as broken, Overall (2015, p. 287) called on the academic community to “focus on correcting the debilitating problems with the academic journal review process.” Our article is a contribution to that correction, by not only developing an understanding of the problems in qualitative researcher experience, but by developing a resource to support the methodological integrity (Levitt et al., 2017) of the peer review of qualitative research (see also Levitt et al., 2021).
The problem is not new. Two decades ago, an experienced medical journal editor guiding scholars new to peer review advised those invited to review a qualitative article: “First, be clear if your expertise is not in qualitative research” (Clark, 2003, p. 228, emphasis added). Clark’s advice reflects a principle now enshrined in discussions of good practice in peer reviewing (e.g., Spigt & Arts, 2010; J. Taylor & Bradbury-Jones, 2014), in ethical guidelines for peer reviewers—such as the Committee on Publication Ethics Council (COPE, 2017) guidelines—and guidance on peer reviewing from publishers (e.g., Sage, n.d.; Wiley, 2024). Wiley’s (2024) guidelines suggest a common driver of negative reviews is reviewers’ lack of awareness of the limitations of their own knowledge and invites reviewers to be honest with themselves about their capacity to provide meaningful feedback. Editors also have an ethical responsibility to select peer reviewers with appropriate expertise, as part of their wider responsibility to ensure a fair and appropriate peer review process (e.g., see Herber et al., 2020; Kleinert & Wager, 2011).
Lack of familiarity with qualitative research (and lack of awareness of the extent of this) can yield methodological incongruence in peer review (Levitt et al., 2017, 2021). In health psychology, for instance, reviewers are often more familiar with postpositivist norms and values than distinctly qualitative traditions (Braun & Clarke, 2023). This can lead to qualitative research not being judged on its own merits, with reviewers critiquing absence of practices associated with, or requesting amendments more suited to, quantitative studies or data analyses, such as testing relationships between variables, providing intercoder reliability scores, validating findings, queries about between group differences, and concerns with sample size, generalizability and (lack of) objectivity (Braithwaite et al., 2014; Herber et al., 2020; Martin et al., 1999). A recent focused mapping review and synthesis of the types of comments commonly made by reviewers on different elements of qualitative articles submitted to medical journals argued that some of the reviewers’ comments suggested not only an “underlying quantitative mindset” (Herber et al., 2020, p. 1) and lack of understanding of the principles underlying qualitative research, but, in some cases, hostility to qualitative research. Even without hostility, misconceptions about the purpose, methods and assumptions of qualitative research, and “attempts to ‘quanti-sise’ qualitative research (apply assumptions of the quantitative paradigms in qualitative assessment)” (Clark, 2003, p. 220) are not just frustrating for authors, they “prohibit publication of their work” (Clark, 2003, p. 220; see also Martin et al., 1999), meaning good quality research gets (unfairly) rejected. The experience, and impact, of this is not equally distributed across qualitative and quantitative paradigms. These problems of a lack of familiarity with qualitative research can be compounded by similar gaps in journal editors’ knowledge, with some editors lacking confidence and skills to guide authors on how to engage with such feedback, particularly when methodological feedback from reviewers is contradictory (Levitt et al., 2017).
Herber et al. (2020) concluded that reviewers unfamiliar with qualitative research should be honest about this lack of knowledge and thus decline invitations rather than merely “apply a quantitative lens in the assessment of a qualitative piece of work” (p. 13). But are these challenges resolved when qualitative researchers provide peer review? The limited research evidence suggests a yes-and-no reply to this question. In the broad area of education research, two reviews from the 1990s offer optimism and caution. In Zaruba et al.’s (1996) analysis of qualitative articles reviewed for The Review of Higher Education they noted that in general, reviewers (who were “generally well versed in the qualitative tradition,” p. 438) “adhered to the ‘culture’ of qualitative research” (p. 454), such as encouraging authors to use methodologically congruent language, appropriate section headings, and break away from traditional reporting formats and not evoking quantitative norms, through requiring quantification of analytic observations or bemoaning the lack of statistical generalizability. However, E. W. Taylor et al. (2001)—also in education—described an implicit set of shared standards about qualitative research (often focused on technical rather than philosophical concerns) evident across peer reviewers. They raised concern about how open the field was to less “traditional” forms of qualitative research and encouraged reviewers (and editors) to “expose themselves to other forms of qualitative research and to stay abreast of what is currently happening in the practice of naturalistic enquiry” (p. 176).
Across the commentaries and analyses of qualitative peer reviews of qualitative research, we noted a tendency to designate certain concepts—such as data saturation (Clark, 2003; Herber et al., 2020), researcher triangulation/coding agreement (Clark, 2003) and member checking (Braithwaite et al., 2014; Clark, 2003; Herber et al., 2020; Zaruba et al., 1996)—as compatible with (all) qualitative research practices, when these practices are contested and/or critiqued by many (e.g., Motulsky, 2021; Varpio et al., 2017, 2021). Universalizing risks methodological incongruence. It also highlights the need for an understanding that congruence is not just a technical consideration, but also an ontoepistemological and conceptual one.

Defining Methodological (In)Congruence in Peer Review

We use (methodological) incongruence to denote peer reviewer and editor comments that do not align with the authors’ research values and methodology, which includes the broader knowledge frameworks within which their work is situated.1 The idea of alignment is regarded by many qualitative researchers to be a key (quality) principle of qualitative research, crucial for producing “sound, meaningful research outcomes” (Chamberlain et al., 2011, p. 152). The concept features in many qualitative quality criteria and reporting standards (Levitt et al., 2017; Tracy, 2010; Yardley, 2024) and is conveyed through the language of methodological integrity (Levitt et al., 2017, 2021), paradigmatic integrity (Hills, 2000), meaningful coherence (Tracy, 2010), and methodological congruence (Pearson et al., 2015). Congruence is often discussed in relation to research design, orienting to the need for alignment between all aspects of research from philosophical orientation through to data generation methods (Hills, 2000). Congruence is also discussed in relation to choices of—for example—paradigm, ontology, or epistemology, which then delimit and direct other choices (Crotty, 1998). There is an emphasis on avoiding drawing on “contradictory and mutually exclusive” aspects of different paradigms (Lincoln et al., 2024, p. 99). This means the ability to achieve congruence effectively depends on a researcher’s grasp of both methods and theory (Yardley, 2024). Methodological incongruence can thus result from (perhaps unknowingly) combining incommensurate frameworks, and concepts and practices rooted in incommensurate frameworks—without any explanation. Tracy (2010), discussing meaningful coherence as one of their eight “big-tent” quality criteria, argued that if a researcher borrows concepts from a different framework, they should “attentively note the disjuncture between the two points of view and explain how [they were] intentionally weaving together the approaches” (p. 849). It is important to acknowledge that qualitative researchers sometimes differ in their dis/agreement with the problematics of paradigm-blending, and some approaches combine different paradigms (e.g., consensual qualitative research, Hill et al., 1997, applied thematic analysis, Guest et al., 2012).
To us, methodological congruence is a necessary guiding principle for reviewers and editors, given the diversity of qualitative research traditions. Good qualitative peer review would involve assessing the overall congruence of what researchers have done and how they have reported it, rather than imposing concepts of rigor specific to particular qualitative traditions. The literature suggests a key risk to congruent peer review comes from universalizing quantitative or qualitative but postpositivist values, concepts, and practices (such as data saturation, member checking, and researcher triangulation, see Varpio et al., 2017) even if they do not fit the underpinning assumptions of many qualitative approaches. This makes some understanding of the philosophical, conceptual, and practice variation in qualitative research necessary for congruent review. There are many different ways to “map” this diversity, but a basic distinction we find useful is between what has been termed small q qualitative and Big Q Qualitative (Braun & Clarke, 2022; Kidder & Fine, 1987). Both small q and Big Q research involves qualitative data generation and analytic techniques, but Big Q deploys these within a framework of qualitative research values. Big Q is conducted within a distinctly qualitative, nonpositivist paradigm (e.g., interpretivist and constructionist), whereas small q often defaults to the postpositivist norms and values that dominate many disciplines (e.g., Varpio et al., 2017), including psychology (Gough & Lyons, 2016; Morrill & Rizo, 2023; Riley et al., 2019). Small q offers a technical or proceduralist orientation, where qualitative approaches are understood as “another set of (technical, rational) tools in the psychologist’s toolbox” (Gough & Lyons, 2016, p. 237). Published small q research rarely includes explicit discussion of theoretical assumptions (Braun & Clarke, 2023). Incongruity happens when standards and criteria that may be applicable to small q qualitative—such as many of those evidenced in the widely used consolidated criteria for reporting qualitative research (COREQ) checklist (Tong et al., 2007)—get (unknowingly) applied to Big Q qualitative, which is judged as wanting, or even as fatally flawed (Braun & Clarke, 2024).

Qualitative Researchers on Incongruous Peer Review

Our research extends the limited literature on the peer review of qualitative research by adding the voices of qualitative researchers. Having expertise in both their own experience and qualitative research, we wanted to know what methodologically incongruent feedback is, what receiving it is like, and how they navigated and responded to such feedback. Through systematically exploring how peer review fails qualitative researchers in this way, we ultimately aim to support the integrity of the peer review process (and qualitative researchers in it) and the methodological congruence of published qualitative research (see also Note 1 of the Supplemental Material).

Methodology


We used a qualitative survey, recruited contributions through wide social media and network sharing, and analyzed the data using a summative-descriptive approach somewhat akin to “codebook” thematic analysis. Ethical approval for the project was granted by the first author’s College of Health, Science and Society Research Ethics Committee.

The Qualitative Survey and Recruitment

The decision to use a qualitative survey was a pragmatic and a participant-centered choice. This was unfunded research. We were mindful of academics’ busy working lives and limited time for research participation, geographic dispersal, and the potentially sensitive nature of the research for some (qualitative surveys offer a strong sense of felt anonymity; Terry & Braun, 2017). We wanted to understand the peer review experiences of qualitative researchers in general, and so to hear from a diverse group of academics with regard to discipline/field, career stage, and qualitative approaches used. The written modality also enabled us to invite contributors to quote from their peer reviews and their responses to these (and many did).
Our short qualitative survey (see Note 3 in Supplemental Materials) asked contributors about the “methodologically incoherent” comments they have received from peer reviewers and editors, and how they addressed them, if they had the opportunity to do so. They were also asked to identify the journals they had received such comments from (we have chosen not to identify these), their discipline/research area, and their career stage (with predefined options of doctoral student, early [within 10 years of doctoral completion], mid [10–20 years], and late [greater than 20 years] career researcher, as well as an option for “other”). We chose to limit the questions asked about contributors to keep the survey as brief as possible (and encourage participation) and to maximize the felt sense of anonymity. Even with these precautions in place, some opted not to provide information about journal names, their discipline/research area and their career stage—some indicated these questions were too identifying, or they wanted to protect the reputation of the journal. We also included an open-ended question for contributors to add anything else they wanted to (which just over half did). We gave contributors the option of being quoted or paraphrased. We also gave them the option of being named in the acknowledgements—just under half opted for this.
We launched the survey on July 7, 2022, and shared via our (work-related) social media accounts, on email lists for qualitative researchers, and through professional networks. After checking the number and content/depth of responses and determining sufficient data had been generated for our purposes, we closed the survey on October 8, 2022.

Contributors

Participation was open to any qualitative researcher who had experienced methodologically incongruent comments from peer review of a qualitative article. We received 163 completed surveys, with just over half (n = 85; 52%) coming from early career researchers (ECRs), 35 (21%) identified as mid-career, 29 (18%) were doctoral students, 10 (6%) were late career; the remaining five (3%) checked another option. These were: undergraduate, postgraduate taught, mid-career but doctorate completed within 10 years, and researchers without a doctorate (n = 2). The preponderance of ECRs and doctoral students may reflect the demographic profile of academics who engage with social media—earlier career academics may be more likely to use social media (Chugh et al., 2021)—and/or how navigating the peer review process intersects with career stage—earlier and later career researchers seem to view and experience peer review differently (e.g., Atjonen, 2019; Horn, 2016; Severin & Chataway, 2021; Watling et al., 2023). Given increasing popularity (and normalization) of qualitative research (e.g., Gough & Lyons, 2016), earlier career academics may also have a greater interest in and engagement with qualitative approaches.
Almost all contributors (151) noted a broad discipline or a specific research focus (many listed multiple research areas). Close to half (75) indicated health, encompassing a wide variety of disciplines (e.g., nursing, public health, medical education); psychology was well-represented (39); education and sociology were the only other disciplines mentioned by 10 or more. Other disciplines/fields included human computer interaction, disability, gender and lesbian, gay, bisexual, transgender, and queer studies, social work, social policy, criminology, management, and sport science. Although not specifically asked about this, from responses it was evident that contributors’ qualitative research experiences and practices crossed the small q and Big Q spectrum.

Data Analysis

Our analytic strategy was similar to codebook thematic approaches (Braun & Clarke, 2022), determined after reading and making notes on the survey responses. We developed categories for reporting the analysis, and then organized and summarized relevant data within these categories. We initially used six categories, but subsequently refined these to four by collapsing three into a broader (universalization) category: (a) inappropriate universalization; (b) strategies for navigating methodologically incongruent comments; (c) power dynamics, loss, and (emotional) labor; and (d) recommendation for improving the integrity of peer review of qualitative articles. Our analytic treatment of the data was broadly experiential. We regarded contributors’ accounts as more or less straightforward representations of their perceptions and experiences (Braun & Clarke, 2013) and wanted to evoke contributors as offering “expert testimony” through granting interpretative authority to them (while also recognizing our inherently interpretative role). This is also reflected in our decision to refer to them as contributors, rather than participants.

Analysis


Most contributors reported widespread experience of methodologically incongruous feedback in peer review. A core concept that drew together different forms of incongruity was universalizing norms or practices associated with quantitative or specific qualitative research approaches. This is conveyed in our first category, inappropriate universalization. Our interest in the implications of incongruous peer review is conveyed in our second and third categories. Our discussion of strategies for navigating these (second category) is contextualized by our exploration of how power structures within academia intersect with responding, and the (emotional) labor around (incongruous) peer review. In lieu of a conventional discussion section, we end by synthesizing contributors’ suggestions for improving the methodological integrity of peer review of qualitative research.

Inappropriate Universalization

Methodologically incongruent reviewing most typically appeared through comments, claims, or expectations that applied standards or practices inappropriate to the article being reviewed. These predominantly reflected postpositivist and/or quantitative research perspectives and standards—but some also reported experiences of inappropriately universalized delimited qualitative standards. Contributors described that in some disciplines (e.g., health, the predominant area contributors worked in), quantitative researchers had begun “dabbling” with qualitative research, approaching it through a postpositivist lens, something evident both in research they produced and their peer reviews (a trend also noted by Riley et al., 2019).
In some cases, peer reviewers and editors seemed to have some understanding that qualitative research is different from quantitative research in some respects, but nonetheless expected qualitative research to conform to the same or similar research values. The most commonly reported modes of universalization related to “sample size,” the need for “reliability practices” (for coding) and (lack of) quantification. For example:
The primary comment I get is about sample size and how it is not “representative,” therefore I cannot make any conclusions from the data. There is such an emphasis on multiple coders and intercoder reliability because it is a quantitative measure, that if you do not have that in your study, the reviewers question your rigour. (C140)
In some contributors’ experience, reviewers and editors treated a constructed-as-small (and therefore unrepresentative, nongeneralizable, nonrandom, not statistically significant; see also Clark, 2003; Herber et al., 2020; Martin et al., 1999; Zaruba et al., 1996) data set as grounds for rejecting an article:
Given the reviewers comment and the small number of highly selected participants on a qualitative review, the manuscript is not considered to be suitable for publication. (C144, quoting an editor)
Although we do publish qualitative studies, we expect all studies to have results that can be generalized to large groups or cultures. It is unlikely that the 15 individuals that you interviewed represent the thousands of immigrants in [country]. (C69, quoting an editor)
Alternatively, reviewers and editors requested that authors collect more data, and/or note a “small sample” as a limitation of the research. We use the term “requested” around changes but often these were framed more fluidly—a reviewer may comment on what had not been done or was missing rather than explicitly stating a practice was required. In the absence of editorial guidance to the contrary, these become effective requests. Some contributors had encountered reviewers and editors requesting power analysis and other calculations to demonstrate validity of their sample size, or evidence of data saturation (i.e., information redundancy) as a stopping criterion (Herber et al., 2020). For example:
On multiple occasions I have been asked to indicate how we achieved data saturation, despite not using Grounded Theory. People seem to think this is the qualitative equivalent of working out sample size and power when using statistics. (C72)
Did you achieve saturation? This important issue needs to be addressed. (C8, quoting an editor)
Requests for measures of the “reliability” of coding were another common example of methodologically incongruous review (see Braithwaite et al., 2014; although Herber et al., 2020 reported this infrequently). For example, C161 described:
Requests for interrater reliability statistics from a secondary coder … they usually want a kappa, gamma or ICC—to supplement in-depth qualitative analyses (e.g., reflexive TA [thematic analysis], IPA [interpretative phenomenological analysis], discourse analysis). … Generalisability vis-a-vis sample size: I am constantly told I need large sample sizes “for generalisability” to conduct qualitative research (hundreds of participants) for designs like IPA, narrative etc. … This is because of the positivistic ontology that plagues Psychology. … They want a table with frequencies/percentages, so they can understand the variability in the themes—which is tough with a study with only 6 people, for example. Or, they want a clear description of “some,” “most,” etc. … They want minimal researcher involvement, standardized codebooks, etc. … They do not understand theoretical frameworks/lenses at all.
In reviewer and editorial feedback such as that reported here, the researcher subjectivity essential to quality (Big Q) qualitative (Gough & Madill, 2012) is conceptualized as a problem and a threat to objectivity, and thus a flaw in the research. Having only one coder—common on qualitative research—was understood by reviewers and editors as problematic. Reviewers and editors wanted authors to add codebooks, rules for coding, additional coders, training of coders, and consensus practices (e.g., multiple coders agreeing codes, multiple researchers agreeing themes/researcher triangulation) to their analytic procedures. Not using qualitative data analysis software (QDAS)—and NVivo specifically—was understood by reviewers and editors as problematic, as these were seen as ways to enhance reliability. Some contributors reported peer reviewers and editors requesting participant validation of the accuracy of transcripts and the use of member checking to ensure the accuracy of interpretations (Zaruba et al., 1996)—again suggesting a conceptualization of researcher subjectivity as a potential source of bias and threat to be contained.
The idea that the world is discretely (and objectively) knowable was conveyed also in numerous reported requests for quantification of the analysis—already noted by C161 (see Clark, 2003). Contributors reported that reviewers and editors wanted (discussion of) frequency counts added to the article or even statistical analysis of the data set. C105 quoted a reviewer as an example of incongruous review they frequently received from reviewers and editors:
I love the work by Braun [sic] on how to do thematic coding, but I wonder why different techniques were not used. For example, a quantitative content analysis would have addressed the frequency of themes and let researchers compare responses across respondent groups (i.e., demographic qualities of respondents).
As this example shows, contributors described reviewers and editors wanting participant demographics to be treated as variables and comparative analyses undertaken (see also Braithwaite et al., 2014; Martin et al., 1999). Contributor C67 quoted an editor who recommended reworking their conversation analysis of a corpus of 25 doctor–patient interactions into a “rigorous quantitative analysis”:
You could identify the patterns of interest, code them in the conversations, and statistically evaluate their occurrence to test against spurious effects. You can of course take an alternate quantitative approach to data analysis, but it needs to be sufficiently rigorous because we can’t trust the conclusions without the numbers and without tests against spurious (chance) patterns.
These various methodologically incongruous requests demonstrate a failure to review qualitative research on its own methodological terms.
These regularly reported methodological incongruous reviewing requests often aligned with practices featured in the popular COREQ checklist (Tong et al., 2007) and other quality and reporting criteria in health sciences more broadly (see Santiago-Delefosse et al., 2016). COREQ was mentioned by some contributors as shaping peer reviewers’ and editors’ (narrow but universalized) views on good practice in reporting qualitative research:
More typically, the reviewer is looking for some key word(s) in the reporting of methods, and most usually the reporting of the analysis. Of course, the words they are looking for (e.g., saturation) may not be relevant to the methodology. … Probably the most common is saturation, e.g., with 10 interviews it’s unlikely you reached data saturation and you have not mentioned it. Sometimes a comment like this is accompanied by reference to the data saturation item from Tong et al.’s checklist (COREQ). The next most common is to be asked for a “coding tree” or “code book” when that is not consistent with the methodology described, e.g., reflexive thematic analysis. Again, this may be accompanied by reference to the item in COREQ that asks for a description of the coding tree. The third most common is to be asked whether transcripts were independently coded, by how many coders, and what measure of coder agreement was used. (C87)
Bigger Q Qualitative contributors tended to view all of the different types of comments described here as problematic (see Morrill & Rizo, 2023). In contrast, (some) smaller q qualitative contributors saw value in COREQ and/or used it to rebut methodological incongruent reviews. They were mostly troubled by the comments of peer reviewers and editors who could not make sense of or did not see any value in qualitative research and wanted qualitative research to look like quantitative research.
Contributors reported comments about research design that universalised postpositivist/quantitative approaches and norms (e.g., representative and generalisable samples, statistical analysis). Many reported encountering assumptions that qualitative research should have hypotheses, or that qualitative researchers should discuss what they expected to find:
It was difficult for me to evaluate how the presented interview protocol was uniquely suited for testing the hypotheses. (C156, quoting a reviewer’s report)
To make the paper more publishable, we would strongly encourage the authors of this paper to consider recruiting a comparison group for a more robust analysis. (C66, quoting a reviewer’s report)
Some noted reviewers and editors commenting that theory should only be used to make empirical predictions. C28 quoted a reviewer:
If they are to use a theoretical framework … it should be one which can confidently make empirical predictions. The current “theory” adds nothing to their paper and, in scientific terms, is not a theory.
Comments like this exemplify our “confidently wrong” characterization of much of the reported methodologically incongruous peer review.
Numerous contributions conveyed peer reviewers and editors confused by and unfamiliar with the style and presentation of qualitative research. Failure to comply with (expected, universalized) norms seemed to render qualitative research puzzling or even incomprehensible (and therefore wrong) to some reviewers and editors, with some unable to ascertain any value in research that was not quantitative. For example, C96 quoted a reviewer’s report which asked “where are the findings? You only provide quotes.” Contributors also reported requests/requirements to change article organization or content to align the presentation of the research with postpositivist/quantitative reporting norms (Tracy, 2012; Walsh, 2015). For example, C9 quoted reviewers reports:
“There is a lot of personal experiences in the method which I’m not sure is necessary to this study” and “Although you are personally involved in the data collection, I think this would read better if it was written in the 3rd person throughout” … “I felt there was an overreliance on quotes to tell their story rather than the author making a strong narrative. To improve consider reducing the quotes used and have more narrative” … “Results sections are generally from the data only. Remove all references to other research in this section. Also leave out reflections on the data for the discussion section.”
This quotation conveys key stylistic aspects noted by many, such as: separating “results” from “discussion,” and removing researchers’ interpretation and references to literature from the former (see also Clark, 2003); removing or reducing data quotations from the “results” (see also Martin et al., 1999); conversely, only presenting data quotations and no analytic commentary in the “results”; writing in the third person; removing (qualitative) “jargon” (terms such as pragmatism); and removing (discussions of) reflexivity, methodology, ontology and epistemology, and other theory.
The contributor’s experiences conveyed a sense in which some reviewers and editors positioned themselves as expert, and as unequivocally right, and the author(s) as wrong, and needing to change:
Being told to write up results and discussion rather than combine these and condescendingly explained what each section should include. … Being told not to use first person as it’s not academic. (C45)
This experience evokes a role more like an examiner than a peer reviewer—something we come back to when we discuss power and emotional labor in peer review. Reviewers and editors’ lack of familiarity with the conventions of reporting qualitative research was sometimes combined with explicit (and implicit) disrespect for, or dismissal of, qualitative research (see also Herber et al., 2020)—usually some version of it being un/less scientific, and idiosyncratic rather than systematic or rigorous. For example:
The paper comes across as particularly idiosyncratic, non-generalisable, and personal level opinion—perhaps from the couch of a psychoanalyst or hypnotist. (C79, quoting a reviewer’s report)
It is just a subjective opinion of the author. It is not in the form of the paper. (C106, quoting a reviewer’s report)
This article certainly can’t be accepted, there is no data here. Author not transparent with equations. (C10, quoting a reviewer’s report)
Some contributors noted this disdain for qualitative research particularly around mixed method research (see also Morrill & Rizo, 2023):
It also doesn’t feel like qual is ever enough on its own right—truly. We need to juxtapose them [quantitative and qualitative] against each other, and then make it known that quant is better, and qual is “supplementing” the analysis. … Reviewers want qual studies written up like quant studies—they want a cookie cutter study that looks like a survey. (C161)
Despite decades of qualitative scholarship, these reviewers appear to still universalize quantitative/postpositivist norms to construct qualitative scholarship as inherently methodologically insufficient. However, inappropriate universalization also featured around qualitative research specifically, when peer reviewers and editors familiar with some qualitative research approaches universalized qualitative research by assuming that the conventions of one particular approach applied to all. Some referred to a kind of “boundary policing” of particular qualitative methods, and how they should be used:
A less frequent problem, but one that does come to mind, is over-confident/over-stated claims about what a “method” can or can’t do. “This isn’t template analysis, because it has themes,” or “This can’t be IPA [interpretative phenomenological analysis], there are two samples,” etc. I think a bit less boundary policing and a bit more curiosity (“It’s interesting to see two samples in an IPA study, can you explain a bit more about how that fits with the approach?”) is all that is needed here. (C124)
I got comments back from the editor saying that I should NOT have piloted my interview because “qualitative research does not involve use of pilots,” and that I should therefore remove reference to this part of the process from the article. (C53)
Universalization of (specific forms of) qualitative research or totalizing declarations produced frustration, and a wish for qualitative researchers to recognize the bounds and limits of their expertise:
I am very tired of qualitative researchers reviewing papers that are outside their expertise and not considering that—e.g., thematic analysts reviewing conversation analysis. For all the complaints qualitative researchers make about quantitative reviewers, surely they would also want to apply that same approach to themselves! (C90)
Those contributors working with approaches that treat language as productive, such as discourse or conversation analysis, appeared to particularly experience this methodologically incongruent form of review:
Reviewers who are obviously unfamiliar with discourse analysis argue that interpretations of discursive functions are over-interpretations and unwarranted claims and suggest to “let the data speak more for themselves.” (C47)
These responses suggest methodological incongruence (and a related issue of qualitative methodological expansion) is perpetuated by qualitative reviewers who do not have a full understanding of the diverse conventions and practices of varying forms of qualitative research, or the diverse philosophical underpinnings and assumptions of different approaches. The kinds of experiences of peer review described suggest the need for “connoisseur” reviewers (Sandelowski, 2015; Sparkes & Smith, 2009), equipped with both expert knowledge and openness and flexibility when encountering unfamiliar methodological approaches. However, having enough reviewers with such expertise—and willingness/capacity to review—remain a challenge.
One contributor requested that we produce a list of reviewer and editor requests that are methodologically incoherent with Big Q qualitative and present this in the article. We have provided this in Note 1 of the Supplemental Material. Such a list was requested because it would be a helpful resource in responding to and rebutting methodologically incoherent feedback. We now turn to the contributors’ strategies for navigating such feedback.

Strategies for Navigating Methodologically Incongruent Feedback

It’s tricky because there is a felt sense that we as authors have to save face for the reviewers/editors even though their comments were methodologically inconsistent. Part of this is about the power differential between the journal, us as authors, and our need and desire to publish our work. Another part is about helping bring editors and reviewers along in a way where they might learn something rather than turned off. (C31)
This quotation powerfully conveys the experience, affect, and power differentials of peer review described by many contributors. Some contributors noted that editors often “shared the reviews with no additional comment or feedback” (C14) and “did not provide guidance” (P122), and so they had to navigate methodologically incongruent and at times contradictory feedback without any editorial support or input:
Reviewer 1 wrote back (paraphrase), “This article certainly can’t be accepted, there is no data here. Author not transparent with equations.” The other reviewer wrote “This is an excellent and rich qualitative study ….” The editor asked me to consider and respond to both reviewers. (C10)
Contributors expressed surprise and disappointment that editors simply “wave through” (C102) reviewers’ methodologically incongruent feedback and demeaning comments about qualitative research; others thought that some editors simply “did not know” (C122) that some reviewers’ comments were incongruent. Some faltered without editorial guidance:
I withdrew the article from the peer review process. I should have spoken to the editor first, but at the time I just felt there was no point surely as the comments were shared with me with no notes about how to engage with them. (C73)
Most navigated through (even contradictory) feedback to resubmit (several had their article rejected). Many outlined strategies for dealing with such feedback, sometimes specific and sometimes general strategies often honed over time and experience (see also Watling et al., 2023). Some selectively ignored methodologically incongruent comments; some opted to withdraw the article when comments illustrated too strong a methodological disconnect (see also Cerejo, 2014). Strategies reported were broadly clustered into (overlapping) practices of educating (including preemptively), seeking support, what might be term “calling out” the decision, acquiescence, and engaging with the editor.
Educating—the most common strategy reported by contributors—typically involved not making the requested changes. It involved explaining why they had not made the requested changes to the article, and the incongruence of the feedback with their particular qualitative approach. Contributors reported citing or quoting relevant methodological literature to support their position, recommending readings for the reviewers, and effectively educating them (and the editor) about the assumptions of qualitative research in general and specific to their approach. The length of time the rebuttal response took was noted by some—contributors described both “lengthy responses” and “blunt rebuttals,” seemingly depending in part on the anticipated receptiveness from the reviewers and/or editor. Much of this was conveyed by C161:
I fight back—I do not give in. I send lengthy responses (several paragraphs) back and hope they will understand. I will explicitly interrogate the claims, and provide references to back up NOT doing power analysis for qual, NOT doing IRR [interrater reliability] for RTA [reflexive thematic analysis], not having 100 participants for an IPA [interpretative phenomenological analysis] study, etc. So, I defend and counter each one of the incorrect arguments, with citations, and this is exhausting, and often times reviewers won’t budge and so I find a new journal that publishes qualitative work.
The process was framed here (and by some others) as a(n exhausting) battle, evoking the adversarial experience noted by others (Jamali et al., 2020). The layers of work hint at the psychological intricacies of the peer review process, discussed (along with power) in the next section. It also connects to something noted particularly by ECRs: the importance of having a good support system in place for responding to reviewers’ and editor’s incongruent comments (see also Watling et al., 2023)—and especially one that encouraged rebuttal:
I was lucky to have a good supervisory team behind me to discuss the comments with. (C151)
It was the other members of the team who are more senior, who reminded me that we could simply politely tell the reviewer no. (C58)
Support typically referred to a “team” around or behind the author and having people “with [a] long history of engagement with qualitative research and well-established reputation in publishing such research” (C163) was notably helpful. This raises questions of how to resource those who do not have access to such support—something we hope this article contributes to.
Another strategy was citing articles from the journal they had submitted to or from similar journals that had used the same methodological approach. Contributors used this approach to support their argument that their article represented established practice in qualitative research:
I also often include citations to recently published articles in the journal I have submitted to, to show examples of interpretive/critical qualitative scholarship that was accepted without compromising their methodological commitments. (C24)
A later career researcher described using disagreement between the reviewers to their advantage, to discredit the methodological incongruent feedback from one of the reviewers. Some contacted the editor directly “to inform them of the problems with these statements” (C90), complain about the review process, or check if the revisions required were deal breakers. Such strategies require a researcher who both knows, and feels able, to do that.
Others described preemptive strategies when writing their article. One ECR noted trying to preempt criticism by explaining in the article why certain practices were not used, citing relevant literature. This strategy did not always work. Another ECR described being asked to do the very thing they had already justified not doing.
Overall, such strategies were sometimes effective, and sometimes not. If pushing back did not work, “caving”/“capitulating,” compromising or submitting elsewhere were the main responses. Some described making a pragmatic decision to comply with reviewer and editor demands because they did not want to revise the article again or have it rejected:
I tend to capitulate to reviewers because I worry the revisions will get sent back again. (C162)
Tried to explain but as an ECR I often feel beholden to reviewers’ comments and pressure to address them. (C84)
This highlights the power inherent in the review process, something potentially more acutely impacting ECRs or inexperienced qualitative scholars. Some noted capitulating previously, when they were less experienced and less confident:
Earlier in my career I just sucked this up and got someone to double code a percentage. Now I refuse and instead include how the team were involved in developing coding frameworks and refining themes from the outset and defend this position epistemologically. (C115)
However, requiring incongruous practices might also “play forward” into what becomes understood as congruent or good practice. One ECR described not just capitulation in the article, but a change in how they taught qualitative methods:
In this case, I very much caved to their demands since I felt the need to get this article published as a tenure track faculty member. Further, reviewer/editor comments such as these have influenced how I teach qualitative methods—I have begun to incorporate a larger focus on interrater reliability statistics/methods to prepare my students for these expectations when trying to publish. (C83)
Others navigated a line between fighting back and capitulation: They partially addressed reviewers’ and editors’ comments, but not so far that they felt they had completely compromised the integrity of their research. For example:
Did not provide interrater reliability but did insert a few words indicating prevalence of sentiment in results (e.g., “Most participants felt …”). (C132)
Some noted their fears of offending the reviewers or the editor when responding to feedback, and the potential implications of this for the publication of their article and their career progression. This takes us to issues of power and (emotional) labor.

Power Dynamics, Loss, and (Emotional) Labor

Challenging editors is very difficult. If I had less fear of implications of offending the editor and peer reviewers, I would email back saying … that I found some of the comments as not showing a thorough understanding of the methodology and analysis (C156).
This section focuses on power dynamics in the peer review system, the impacts of this, and the emotional and other labor participants engaged in as they navigated through it. Earlier quotations have already evoked a system in which authors feel comparatively powerless, something we noted felt to us more like an examination than a peer engagement dynamic. For some, this power dynamic also reflected disciplinary/scholarly failure to accord qualitative approaches status equal to quantitative:
I think it is the height of arrogance that people who are not familiar with this field feel they can review it. I feel it stems from an attitude amongst some disciplines and fields that qualitative research is somehow “less than” and does not have its own rigorous methodology. (C78)
Previous quotations illustrated how the language and framing of qualitative research in peer review situated it as lesser than quantitative, as nonrigorous. Some noted that methodologically incongruent comments went hand-in-hand with, or were a veil for, other types of poor peer review practices, such as subtle racism. For example, research with an Indigenous community was characterized by a reviewer as “rather parochial” (C141). Another contributor noted Global North/South power dynamics at play:
Based on my experience and the experiences of colleagues who conduct qualitative research in the “Global South” and submit papers to “Global North” journals (even the critical ones), this is also worth noting. Often the “methodologically incoherent comments” we receive are cloaked in subtle and overt tones of intellectual superiority: write this sentence this way and not that way, cite research and methods from Global North researchers and not local sources (which may be more relevant to the subject at hand). (C143)
These accounts evoke the disproportionate harms of peer review on researchers from underrepresented groups noted by others (e.g., Rodríguez-Bravo et al., 2017; Silbiger & Stubler, 2019). Many contributors noted impacts of methodologically incongruous review. Inappropriate review comments had emotional impacts—particularly for ECRs (see also Majumder, 2016; Watling et al., 2023). C156 was “completely floored by these comments”; C30 noted they felt:
Miserable and with a decline in confidence, whereas my already raging impostor syndrome flourished.
Describing the labor of responding, some referenced the importance of tone in responding to such comments: noting practices of responding “politely but firmly” (C142), and being “kind and firm” (C87), “patient” (C99), and “respectful” (C122), and using “‘appeasing’ language” (C85) and a “professional and friendly tone” (C146). Cohering with existing reports of the psychological burden of peer review (e.g., Horn, 2016; Majumder, 2016; Watling et al., 2023), some contributors reported feeling tired and frustrated with methodologically incongruous reviewer and editor comments. Later career researchers expressed concern about the damaging psychological impact of such comments on ECRs (noted in the wider literature; Hollister et al., 2023). These types of comments made some ECRs question continuing with a particular type of qualitative research or qualitative research in general:
It’s very frustrating and makes me reluctant to keep doing co-production work in future because it’s so often an uphill battle with reviewers. (C162)
I think this is a serious issue—when early career researchers receive this sort of advice it can be highly demoralising especially. (C14)
Some ECRs’ concerns went beyond frustration, as they considered the impact of such reviews, and the concomitant lack of knowledge of qualitative research in their disciplines, on their career progression. Contributors noted the time involved in rebutting methodologically incongruent comments—time that could be better and more productively spent elsewhere. Some ECRs felt like they were missing out on opportunities for feedback, learning and development, for intellectual dialogue or improvements to their article, because of poor peer review practices (see also Watling et al., 2023). For example, C94 noted:
I’m sure my own understanding of qualitative research methods could improve, but it is frustrating to be having the shallow, basic arguments via rebuttal rather than much more interesting and enriching arguments.
Publication obligations created extra pressures for ECRs, which can produce anxiety when receiving negative peer reviews (Horn, 2016):
On several occasions I did not have the opportunity to respond to comments that seemed extremely unfair and jeopardized my career. It takes so long to respond to such comments, especially at an early career stage. It has been hard for me personally to remain motivated in my profession as a qualitative researcher. (C146)
As an early career researcher, I have twice recently been told by reviewers that my sample, for qualitative papers, is not statistically significant. It is absolutely gutting to know your professional career is being determined by people who either don’t respect or understand it. (C33)
Some reported “having to educate” (P127) longer serving/more experienced reviewers and editors about qualitative research:
It is stressful and frustrating to need to teach reviewers about qualitative methods (when they agreed that they had the methodological expertise to review the paper in the first place), and disappointing when the editors also don’t know any better or use these reviews as an excuse to reject the paper. (C144)
Peer review is good when it’s good, but it’s often bad because you’re basically just teaching Qual 101 to reviewers/editors and it’s extremely boring. I don’t find that my work is improved or made clear, but rather that I’m engaging in labour for people who shouldn’t be reviewing my work. … It’s just extremely disappointing as a PhD/early career researcher that peer review often doesn’t involve people engaging seriously with your ideas or the literature you are speaking to. Instead, you’re just handholding some reviewer through the basics of qualitative research and everyone’s time is wasted. (C8)
One contributor expressed “dramatically” how the experience of incongruous peer review made them feel—in light of the real world consequences of challenges to the validity and integrity of qualitative research through methodologically incongruous peer review:
I’m aware of how dramatic this sounds, but the reality of reviewers not understanding this methodology and analysis means I think I’ve wasted my time doing a PhD. … If I can’t publish, it will be incredibly difficult to secure a permanent position. I strongly believe in the value of this research, but I wish I could go back in time and just not do the PhD. (C156)
Within this system, some ECRs reported that they felt obligated to comply with methodologically incongruent requests because they needed to get published:
My status as a tenure track faculty member also makes me feel pressured to get published and comply, even if I don’t agree with the reviewer requests. (C83)
This knowing compromise (see also Overall, 2015) highlights unequal power and evidences how peer review can do the opposite of what it is intended for and work against quality. For others, the work to continue to do and publish qualitative research with methodological integrity was paramount—the language used by C161 conveyed the effort it can take:
I feel alone in my department, and I am constantly fighting. … I’m passionate in qual, and will continue to fight—I will die on this hill!!!!
This echoes Morrill and Rizo’s (2023, p. 416) call for qualitative researchers “to hold steadfast to preserve methodological pluralism and the transformative possibilities of qualitative paradigms to resist assimilation, misappropriation, and co-option.” Although we agree, our research demonstrates the burden this can take. Across the quotations already presented, contributors’ frustrations are evident, as is the affective impacts of, and labor required to respond to, such review. C78 evoked this starkly:
I am so tired of receiving comments like these. This is not the only occasion I have had these, just the latest.

Recommendations for Improving the Integrity of Peer Review of Qualitative Articles


Here, we synthesize contributors recommendations for improving the methodological integrity of the peer review of qualitative research (which sometimes mapped onto existing recommendations; e.g., Braun & Clarke, 2023; Levitt et al., 2018). We have categorized suggestions into the responsibilities of: (a) journals; (b) editors; and (c) reviewers.
Some suggested that journals/editorial boards should clarify in the journal “values” statement (aims and scope) for their journal whether qualitative research is within the scope, and if it is, which types of qualitative research are within that scope. There were mixed perspectives on journals’ use of reporting guidelines or checklists—reflecting different positions more widely (e.g., Allen et al., 2019; Barbour, 2001; Braun & Clarke, 2024; Buus & Perron, 2020; Herber et al., 2020; Spigt & Arts, 2010). Caution was expressed about checklists (etc.):
As a means of quality evaluation [they] are a bit limited, and they do tend to shy away from the key conceptual questions that an experienced and knowledgeable reviewer would bring. (C124)
Some argued that many checklists evidenced the dominance of postpositivism in health research (Buus & Perron, 2020) and wanted editorial boards/editors to recognize the limited applicability of checklists like COREQ (Tong et al., 2007) and the Critical Appraisal Skills Programme (2018). One later career researcher expressed concern that some editors might respond to this article by mandating the use of checklists or reporting guidelines by peer reviewers—which they felt was not the answer. More beneficial would be ensuring that editorial boards and reviewer pools include qualitative experts. And either an associate editor or reviewer with expertise in qualitative methodology should peer review—with a methodology focus—qualitative articles, alongside other reviewers (see also Cooper, 2009; Herber et al., 2020; Spigt & Arts, 2010).
A key request for editors was to reject an “any reviewer will do” (C17) position and select reviewers with appropriate methodological expertise (see also Herber et al., 2020), including screening reviewers for such expertise (see Cooper, 2009; Majumder, 2016). Contributors referred to a need for “standards of competence” (C84) for peer reviewers, and training for peer review (widely recommended in the literature; e.g., Cooper, 2009; Majumder, 2016; Overall, 2015). The other key recommendation for editors was that they should intervene when peer reviewers make incongruent comments—not only over-ruling reviewers, and advising authors to ignore incoherent comments, but also feeding back to and educating reviewers.
More care should be taken by editors taking on comments from non-qual or specific tradition reviewers. I get reviewing is hard but editors need to be stronger in stating that suggestions of saturation or generalisability by some reviewers is just plain wrong—and tell reviewers this!!! Only way they may be informed. We can’t rely on reviewers being humble enough to recognise their methodological weakness or to change perceptions of some that qual is just asking questions and writing sneers on a paper. (C127)
One participant who was an editor described engaging in exactly this type of intervention:
I always ensure that at least one of the reviewers is a qualitative researcher for such papers. I always overule quantitative researchers who incorrectly criticise qualitative methods if their comments are not correct. I tend to also message them privately to explain why the comments are not correct in the hope that this helps them to improve their review of qualitative papers in the future. (C118)
This recommendation for greater editorial intervention in poor peer review has been noted elsewhere (Cooper, 2009; Hollister et al., 2023).2
Regarding reviewers themselves, some suggested that if peer reviewers do not have appropriate methodological knowledge, they should confine their comments to the subject matter, or not accept the invitation, which aligns with existing ethical/good practice guidelines for peer review (e.g., COPE, 2017; Napolitani et al., 2017; Spigt & Arts, 2010). Reflecting honestly on the limitations of their knowledge and expertise was also recommended for reviewers (see also Herber et al., 2020):
If you are not a qualitative researcher and don’t understand the nuances of various qual approaches then either don’t review qual articles or at least acknowledge this in your reviewer feedback and don’t comment on something you don’t know about. (C101)
Contributors’ suggestions for improvements were sometimes couched in acknowledgement of the challenges with implementing changes—including the voluntary and unpaid nature of peer review and journal editing in many fields, the increasing difficulty editors experience in securing reviewers (let alone ones with appropriate knowledge; Overall, 2015), and the limited potential pool of reviewers with deep wells of qualitative knowledge. However, the impacts of not implementing changes—on individuals and on scholarship more broadly—suggests an urgent need for better qualitative reviewing practices.

Conclusions


The poor peer review practices described by contributors, particularly with regard to the universalizing of postpositivist assumptions and norms, evidence Morrill and Rizo’s (2023, p. 404) argument that “qualitative methods have been constrained by scientism and subsumed within postpositivist frameworks” (see also Gough & Lyons, 2016). They noted that publishing procedures may restrict diversity in qualitative research. Our study evidences the contribution of peer review to this phenomenon, and the way reviewers and editors unfamiliar with (different approaches to) qualitative research, and publication pressures, can compromise the methodological integrity of qualitative research. Morrill and Rizo argued for methodological integrity “as a radical tool against co-option” (p. 404). Our study evidences that many qualitative researchers do indeed push back against methodological incongruent comments, but not always successfully. Methodologically incongruous review sometimes acted as a tool for gatekeeping knowledge production, with the reviewer and editor defining the right way to do research (see Atjonen, 2018). Peer review offers the potential for learning and skills building. Yet the experiences the contributors reported often strayed far from the constructive feedback—feeding forward (Atjonen, 2018)—discussed in ethical and good practice guidelines for peer review (e.g., Allen et al., 2019; COPE, 2017).
That the kinds of methodological incongruent comments described by our contributors echo those documented from the late 1990s/early 2000s (e.g., Clark, 2003; Martin et al., 1999; Zaruba et al., 1996), and since (e.g., Braithwaite et al., 2014; Herber et al., 2020), highlights the persistence of poor review of qualitative research over several decades, suggesting frustratingly little has changed, even recently as qualitative research has become more widely valued and used. We call on peer reviewers and editors to take seriously the ethical principles and responsibilities of peer review and consider the contributors’ suggestions for improving the integrity of the peer review of qualitative research. And, we call on publishers3 to support reviewers and editors in this undertaking and to develop systems of peer review in which the methodological integrity of peer review, and the quality of all forms of qualitative research, can flourish.