MapOSR - A mapping review dataset of empirical studies on Open Science

Data collection protocol

We designed the study as a mapping review. Our aim was to identify all empirical studies concerned with Open Science or any of its key elements, to be used as a basis for deeper investigations. Research on Open Science and its concrete concept varies, and the term “Open Science” is not new. However, Open Science as a movement of new research practices enabled by means of technical innovations on the Internet, has been discussed for over twenty years (Bartling & Friesike, 2014). Following the Foster taxonomy of Open Science, mapping in this study covered research related to key elements of Open Science (Open Data, Open Access, and Open Source) which also guided our search strategies. Our aim was to map research investigating these key elements of the Open Science movement, and not to identify the extent of Open Science concepts discussed, as a scoping review might aim for (cp. Grant & Booth, 2009). Moreover, in this first step, we did not synthesize any results like in a systematic review, but annotated the publications with five key features to give a better overview of the nature of the studies. With these settings and restrictions, we consider our study as a mapping review of empirical studies on defined Open Science elements.

Literature search and screening

Considering the recommendations on literature reviews (Gough et al., 2017), we carried out a systematic search, included and excluded publications based on factual criteria, and annotated the relevant publications to characterize main study design and the key features regarding the covered Open Science aspects, study method, disciplinary focus, targeted group and geographical scope. We did not pre-register the review because the idea was developed out of a research group that first collected Open Science studies, and then went on to expand the work with a systematic search and annotation. This first snowball search went over a period of six months. Researchers made an announcement on Twitter and researchgate.net in June 2020 and invited colleagues to contribute to the collection of empirical studies on Open Science. The results were included in the project’s publicly available Zotero-Library². The first entry was made on 30 June 2020, the last on 16 March 2021. The snowball search yielded 126 publications.

In addition, we conducted a systematic literature search on January 26th and 27th, 2021. We searched in the Web of Science (all indices) and Scopus databases. The search query consists of two blocks: a) terms of the Open Science elements, b) terms describing any empirical study (see Table 1).

We deliberately excluded terms relating to open education, like open educational resources (OER) and open educational practices. Although OER are mentioned as part of Open Science (see e.g. FOSTER), research on them as well as educational practices span a different research field quite separated from discussions on Open Science (Scanlon, 2013). Including OER and similar research would therefore have resulted in a very large corpus, which was beyond the scope of this study. Similarly, we excluded citizen science from our search. Block b was necessary to limit the retrieved publications to a manageable number, as block a alone would have resulted in a large number of non-empirical studies discussing Open Science. After testing several search strings and checking the results, we decided to search Open Science elements in the title field only. Terms describing empirical studies were searched in title, abstract, and (author) keywords. Additionally, we limited results to the document types article, book, book chapter, and proceedings paper. As the term “Open Science” is rarely mentioned in the research literature before 2000 (Blümel & Beng, 2018), the date range was specified from 2000 to 2020. This search yielded 3651 publications. Table 1 shows the original queries for the Web of Science and Scopus.

The snowball search and the systematic search in the two databases resulted in 3777 publications. From these, we removed 842 duplicates (see Figure 1). The titles and abstracts of the remaining 2935 publications were independently screened by three coders (all authors of this study) according to the following inclusion criteria:

Figure 1. PRISMA Flowchart depicting the search and selection process based on Page et al. (2021).

Download as a PowerPoint slideDownload as a PowerPoint slide

Disagreement was resolved through discussion. The screening resulted in 2101 publications being excluded.

Annotating key features

The annotations of key features followed several iterative steps. The 834 publications were coded along five categories as described in the codebook (see Table 2 and mapOSR_codebook_V4.csv in the data repository, Extended data, Lasser et al., 2022) Action, method, discipline, group and geo scope. Within all categories, several labels could be awarded at the same time. We distributed the publications randomly across nine coders. These coders were trained in the codebook through joint development, refinement, and discussion in two rounds of coding.

During the coding process, we excluded another 139 studies that did not meet the inclusion criteria upon closer inspection, e.g. for some of the studies the empirical design was not clear. Most exclusions resulted mainly from duplications between conference papers and corresponding publications. The final sample of coded publications therefore included n=695 publications. We adapted the codebook during our process with regard to the following aspects: In the action category we assessed which aspect of Open Science was targeted in the publication. We adapted the labels within the category based on the FOSTER taxonomy and added ‘open education’ and ‘open participation’ (categories are explained in Table 2). We note that while we did not include Open Education and Open Participation in our database search, we still included them in our codebook, to leave room for future extensions of our approach to these categories. Furthermore, we converted ‘open reproducible research’ into a broader ‘open methodology’. The second category describes the methods that are applied to empirically study the chosen aspect of Open Science, such as bibliometric studies or surveys. In the third category we coded the disciplines that are targeted with the study, such as engineering or social sciences. The selection of labels for this category is based on the OECD-Frascati Manual (OECD, 2015). The fourth category describes the group under investigation, such as researchers or librarians. In the last category we recorded the geographical scope of the empirical study according to design and included cases. The labels in this category were based on the ISO 3166-1 alpha-3 codes for countries.

Interrater reliability

Interrater agreement was calculated for each label within the five categories of the codebook. For this purpose, we double-coded 63 of the 697 publications (9%). The coders were evenly distributed across the data underlying the computation of the interrater agreement. The occurrence for several of the dichotomous labels (dummy transformed from the categories) was strongly imbalanced. An example of this is the occurrence frequency of certain countries in the geo category that were never or very rarely coded. Cohen’s kappa, the standard measure of agreement for dichotomous categorical variables, leads to biased values for skewed variables and was therefore not appropriate in this case (Xu & Lorber, 2014). We therefore resorted to simple percentage agreement values for all labels.

Based on the results, we adapted category labels for geo and discipline, i.e. we recoded the labels of “geo=none” and “geo=all” to “geo=unspecific”, and “discipline=none” and “discipline=all” to “discipline=unspecific” again due to the skewed distribution. A reason for the coders’ disagreements for these category labels was that empirical studies do not always explicitly state their geographical or disciplinary focus. For example, bibliometric studies usually investigate publications from determined journals. Here, some coders labeled “geo=none” or “discipline=none” as they did not deviate geographical or disciplinary focus from a journal sample. Other coders annotated “all” to the categories for the same reason, i.e. the journal sample does not deliberately limit geo or discipline in any way.

We calculated the percent agreement for each of the 36 labels from the five categories that were double coded. In this section we are only reporting a summary of the agreement (see Table 3); details for the data transformation, recoding, and results are reported in the documentation (see file “reliability.html” in the code repository; Lasser & Schneider, 2022).

Risk of bias and limitations

The following limitations should be considered with any use of the data set: despite the snowball search, which led to relevant results for the mapping review, we only did the systematic search in two databases, due to time constraints. As Web of Science and Scopus do not include all research literature and are biased towards specific criteria like publications (journal articles), languages (English-focused), and journals that are published in the United States, we lack other relevant peer-reviewed publications not covered in the two databases. We did not explicitly search for further gray literature to complement the results from the database search, therefore our data set may be susceptible to publication bias. Furthermore, in the inclusion criteria, we specify English, German, Italian, French or Spanish as the languages of publications due to the languages skills of the authors and coders involved in our study. This systematically excludes publications in other languages and thus regions investigated. Also, the terms used in the search query were not translated to German, Italian, French, Spanish. Therefore, the database search only returned publications in these languages if an abstract or title was available in English. We invite native speakers of other languages to apply the selection criteria and coding system to other databases and searches in their language and thus contribute to the expansion of the data set.

Underlying data

Zenodo: MapOSR - A Mapping Review Dataset of Empirical Studies on Open Science, https://doi.org/10.5281/zenodo.6491891 (Lasser et al., 2022)

This project contains the following underlying data:

Extended data

PRISMA checklist and flow chart for”MapOSR - A Systematic Mapping Review of Empirical Studies on Open Science” are deposited on Zenodo: https://doi.org/10.5281/zenodo.6491891

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Analysis code available from: https://github.com/JanaLasser/mapping-open-science-research/tree/v1.0

Archived analysis code as at time of publication: https://doi.org/10.5281/zenodo.6491829

License: MIT