Alaina L. Pearce
Assistant Research Data Management Librarian
Penn State University Library
http://orcid.org/0000-0003-3157-6566
http://orcid.org/0000-0003-3157-6566
Introduction
Over the past two decades, there has been a renewed focus on transparent or ‘open’ science. While researchers generally endorse the importance of transparent and reproducible research, adoption of open practices has been slow. This is due, in part, to the fact that open science practices require substantially greater time, effort, and resources than traditional research workflows (Hostler, 2023), and these efforts have not been valued in review, promotion and tenure (RPT) processes (Alperin et al., 2022; Schimanski & Alperin, 2018). As the ‘reproducibility crisis’ and open science movement gained broader public awareness, federal funding agencies developed open access and data sharing policies in an effort to increase public trust in science (Holdren, 2013; Nelson, 2022; Thibault et al., 2023). Many researchers have viewed these new sponsor requirements as unfunded mandates due to added burden, lack of clear individual incentives, and still evolving data infrastructure (Borgman & Bourne, 2022).
Despite clear increases in time, effort, and resources needed to meet new federal open data policies (Gownaris et al., 2022; Hostler, 2023; Wilhelm et al., 2014), there have been minimal changes to academic RPT prioritization of publications and citations (i.e., ‘publish or perish’; Schimanski & Alperin, 2018). While data librarians have advocated for recognition of published datasets and one third of faculty believe data should be valued equally to publications (Schimanski & Alperin, 2018), these efforts have yet to alter RPT processes at most academic institutions. This creates tension for researchers who must prioritize RPT expectations while navigating open data policies (Makula, 2024). To make data sharing and open science more appealing and sustainable, it is important to show researchers how they can capitalize on existing incentive structures and publication models to get credit for sharing data. This editorial will introduce federal open data policies and the Data as Scholarship framework, which aims to provide research data management (RDM) training while addressing the incentive gap for open science.
Federal Open Access Policies
With goals of increasing trust in science and maximizing return of taxpayer investment in research, in 2022 the Office of Science and Technology Policy issued a memorandum requiring federal agencies that provide research funding to build upon existing public access policies to ensure research data are made freely available to the public at time of publication (Nelson, 2022). While the National Institutes of Health was the first to implement their policy (Final NIH Policy for Data Management and Sharing, 2023), all agencies were required to release their final policy by December 31st, 2025 (however, some agencies have not yet released their final policies). As part of these new policies, researchers will be required to include a data management and sharing plan (DMSP) for all new research grants and competitive renewals. While there are differences across agencies, all agencies require researchers to describe the data and associated metadata expected to be generated from their proposed research along with timelines for preserving, accessing, and sharing data and metadata.
Data as Scholarship Framework
The Data as Scholarship framework is a guide for developing RDM workshops that shift the framing away from sponsor compliance to career advancement. To do this, it is important to: 1) acknowledge unseen costs of open data; 2) focus on ‘good enough’ RDM skills; and 3) identify strategies to leverage existing RPT expectations so data sharing results in scholarly outcomes currently valued by academic institutions.
Unseen Costs of Open Data
The additional time, resources, and effort required for open data practices is often overlooked in policy discussions (Hostler, 2023). An analysis of costs related to sharing clinical research data found four primary areas of increased cost: 1) infrastructure and administration; 2) standardization; 3) human resources; and 4) opportunity costs (Table 1; Mohr et al., 2024; Wilhelm et al., 2014). While data infrastructure is growing and the focus of many federal and institutional policy initiatives, other data sharing related costs are overlooked since they relate to activities require to conduct research regardless of data sharing such as staffing and training (see Table 1). However, the cost and effort required to organize and document data for others to use and understand is substantially greater than what is required for maintaining high quality data for personal or lab use (Wilhelm et al., 2014). A recent analysis by the Association of Research Libraries showed these added data sharing related costs accounted for an average of 6% of researchers’ grant budgets (Mohr et al., 2024). While the NIH has allowed budgeting for these costs, there has been no change in overall budgets limits. Therefore, dedicating resources for data sharing costs takes resources away from other areas of the project.
| Table 1. Unseen Costs of Open Data | |
| Sector | Added Costs |
| Infrastructure and Administration | Ethical and regulatory complianceFunder compliance |
| Standardization | Documentation and standardization of data pipeline from collection to processing and analysisGeneration of metadata and data documentation guidelinesCreation or implementation of a data standard |
| Human Resources | On-going RDM trainingAdditional time to implement data standards and generate metadataQuality assurance and quality improvement |
| Opportunity Costs | Time lost for activities that are prioritized by RPT:PublicationsGrant proposals |
| Information aggregated and summarized from Mohr et al., 2024; Wilhelm et al., 2014 | |
‘Good Enough’ Research Data Management Practices
RDM is critical to managing and publishing open data; however, researchers often learn these skills via trial-and-error or inherited lab processes rather than formal instruction in training programs. This leads to gaps in RDM skills and/or practices that produce more errors and less transparency. The need for additional RDM training to comply with sponsor open data requirements further strains limited researcher time and resources (Dempsey et al., 2022; Wilhelm et al., 2014). Therefore, rather than focusing on technical skills or advanced RDM, this framework recommends focusing on ‘good enough’ data practices to meet researchers where they are in terms of RDM skills and the time and effort they can dedicate.
‘Good enough’ practices need to be relatively low effort, have a shallow learning curve, have immediate and future benefits, and increase data reusability. While not an exhaustive list, see Table 2 for ‘good enough’ RDM practices that have been tested in the development of this framework; these can be viewed as an outline for a single workshop or as components that can be mixed and matched to create tailored curriculum. Thus far, these ‘good enough’ practices have been presented to researchers at various stages from graduate students through mid-career faculty as part of stand-alone workshops, bootcamps, and choose-your-own-adventure style of training driven by audience interest. Feedback indicates attendees are motivated to implement these practices and often request additional, in-depth workshops on various RDM components. Therefore, focusing on ‘good enough’ RDM in this framework offers practical, immediate actions researchers can implement to make their data open.
| Table 2. Example ‘Good Enough’ RDM Practices | |
| Practice | Objectives |
| Preserve Raw Data | Keep a back-up copy of un-processed, raw data that to maintain a ground truth (if data size is not prohibitively large) |
| Create a Central Hub | Introduction to hierarchal directory structuresDiscuss the benefit of replicable or common structures across projectsShow how by using abstract upper-level directory names (e.g., ‘admin’, ‘data’, ‘protocol’) |
| Use Meaningful Names | Best practices for human- and computer-readable filenames Discuss common conventions: snake case, camel case, kabob caseIntroduce concept of a data standard or commonly agreed upon naming convention for a project and/or lab |
| Preserve the Journey | Introduce concepts of version control and provenanceApproaches for manual version controlIntroduction to options for more advanced, automatic version control |
| Avoid Manual Manipulations | Discuss how manual data manipulation can limit transparency of dataIntroduce how/where to document manual processesExample: SPSS paste function to get syntax for manual processes |
| ‘Tidy’ Your Data | Introduce the basic organization of data expected by most statistical softwareVariables in columnsObservations in rowOne datum represented in each cell |
| Metadata Magic | What is metadata?README filesData dictionaries |
Leveraging Existing Incentive Structures in RPT Processes
While data librarians and projects like Make Data Count recognize published datasets as primary research outputs, most academic institutions’ incentive structures do not value open data as highly as more traditional research outputs. Analysis of RPT review committee criteria shows there is increasing reliance on quantitative markers of research productivity such as number of peer-reviewed publications and citations (Alperin et al., 2022; Schimanski & Alperin, 2018). When reviewing materials (e.g., formal policy, faculty handbooks, presentations) related to 129 different Universities’ RPT processes, data or data-related outputs were only mentioned in 16% of RPT documents for R-type institutions, with the percentage dropping for less research-focused institutions (Alperin et al., 2022). Although about a third of faculty believe data should be credited equally to peer-reviewed publications (Schimanski & Alperin, 2018), it is unknown how committees value data-related outputs even when mentioned in RPT documents. Given the slow, bottom-up (researcher/department driven) push for RPT changes, researchers need to leverage the value of peer-reviewed publications and citations to get full recognition for the time and effort require to make data open.
Data papers are peer-reviewed publications accompanying open data that provide a publication and direct mechanism for citing shared data. Elsevier, Wiley, Springer, and Frontiers, in addition to many other publishers, have launched data journals and/or a mechanism to publish data papers within existing publications over the past decade. While making data openly available along with a published research articles is associated with a greater number of citations (Leitner et al., 2016), data papers may amplify this effect as authors tend to cite both the primary research paper and the data paper (Fu et al., 2023). In addition to gaining an additional peer-reviewed publication and citations, data papers have the benefit of alleviating researcher concerns of data misuse or misinterpretation (Gownaris et al., 2022; Hendriks et al., 2022) by providing a full description of study methods, data processing and quality, and important considerations (Jiao et al., 2024).
Conclusion and Implementation
Data as Scholarship is a flexible framework that can guide programing that improves RDM, encourages data sharing compliance, and helps researchers see and achieve strategic benefits of data sharing. This framework has been successfully implemented in a 3-part workshop series and a condensed single-session format. Feedback from attendees emphasized the immediate practicality of ‘good enough’ RDM and indicated excitement at the idea of authoring a data paper. The excitement about data papers was particularly common among graduate students, postdocs, and early career researchers who are still working to build their CV and compete in competitive tenure-track job markets.
Key to the Data as Scholarship framework is approaching researchers and framing workshops with three primary goals: 1) acknowledging additional strain researchers are facing, particularly in uncertain funding climates; 2) focusing on practical, immediately implemental ‘good enough’ RDM skills; and 3) showing researchers how to leverage existing incentive systems so data sharing results in scholarly outcomes currently valued by academic institutions.
References
Alperin, J. P., Schimanski, L. A., La, M., Niles, M. T., & McKiernan, E. C. (2022). The Value of Data and Other Non-traditional Scholarly Outputs in Academic Review, Promotion, and Tenure in Canada and the United States. In A. L. Berez-Kroeker, B. McDonnell, E. Koller, & L. B. Collister (Eds.), The Open Handbook of Linguistic Data Management (pp. 171–182). The MIT Press. https://doi.org/10.7551/mitpress/12200.003.0017
Borgman, C. L., & Bourne, P. E. (2022). Why It Takes a Village to Manage and Share Data. Harvard Data Science Review. https://doi.org/10.1162/99608f92.42eec111
Dempsey, W. P., Foster, I., Fraser, S., & Kesselman, C. (2022). Sharing Begins at Home: How Continuous and Ubiquitous FAIRness Can Enhance Research Productivity and Data Reuse. Harvard Data Science Review. https://doi.org/10.1162/99608f92.44d21b86
Final NIH Policy for Data Management and Sharing (Notice No. NOT-OD-21-013). (2023). National Institutes of Health. https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html
Fu, J., Tian, L., Zhang, C., & Li, J. (2023). Opening research data contributes to the citations of related research articles: Evidence from Data in Brief. Learned Publishing, 36(3), 426–438. https://doi.org/10.1002/leap.1551
Gownaris, N. J., Vermeir, K., Bittner, M.-I., Gunawardena, L., Kaur-Ghumaan, S., Lepenies, R., Ntsefong, G. N., & Zakari, I. S. (2022). Barriers to Full Participation in the Open Science Life Cycle among Early Career Researchers. Data Science Journal, 21, 2. https://doi.org/10.5334/dsj-2022-002
Hendriks, S., Ramos, K. M., & Grady, C. (2022). Survey of Investigators About Sharing Human Research Data in the Neurosciences. Neurology, 99(12). https://doi.org/10.1212/WNL.0000000000200886
Holdren, J. P. (2013). Memorandum for the Heads of Executive Departments and Agencies: Increasing Access to the Results of Federally Funded Scientific Research. U.S. Office of Science and Technology Policy (OSTP), Executive Office of the President. https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/ostp_public_access_memo_2013.pdf
Hostler, T. J. (2023). The Invisible Workload of Open Research. Journal of Trial and Error, 4(1), 21–36. https://doi.org/10.36850/mr5
Jiao, H., Qiu, Y., Ma, X., & Yang, B. (2024). Dissemination effect of data papers on scientific datasets. Journal of the Association for Information Science and Technology, 75(2), 115–131. https://doi.org/10.1002/asi.24843
Leitner, F., Bielza, C., Hill, S. L., & Larrañaga, P. (2016). Data Publications Correlate with Citation Impact. Frontiers in Neuroscience, 10. https://doi.org/10.3389/fnins.2016.00419
Makula, A. (2024). And Never the Twain Shall Meet? Institutional Open Access Policies (IOAPs) and Review, Promotion, and Tenure (RPT). Journal of Librarianship and Scholarly Communication, 12(1). https://doi.org/10.31274/jlsc.16899
Mohr, A. H., Carlson, J., Ge, L., Herndon, J., Kozlowski, W., Moore, J., Petters, J., Taylor, S., & Vitale, C. H. (2024). Making Research Data Publicly Accessible: Estimates of Institutional & Researcher Expenses. Association of Research Libraries. https://doi.org/10.29242/report.radsexpense2024
Nelson, A. (2022). Memorandum for the Heads of Executive Departments and Agencies: Ensuring Free, Immediate, and Equitable Access to Federally Funded Research. U.S. Office of Science and Technology Policy (OSTP), Executive Office of the President. https://www.whitehouse.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf
Schimanski, L. A., & Alperin, J. P. (2018). The evaluation of scholarship in academic promotion and tenure processes: Past, present, and future. F1000Research, 7, 1605. https://doi.org/10.12688/f1000research.16493.1
Thibault, R. T., Amaral, O. B., Argolo, F., Bandrowski, A. E., Davidson, A. R., & Drude, N. I. (2023). Open Science 2.0: Towards a truly collaborative research ecosystem. PLOS Biology, 21(10), e3002362. https://doi.org/10.1371/journal.pbio.3002362
Wilhelm, E. E., Oster, E., & Shoulson, I. (2014). Approaches and Costs for Sharing Clinical Research Data. JAMA, 311(12), 1201. https://doi.org/10.1001/jama.2014.850
