Helping Canadian Institutions Evaluate their Research Data Management Practices

Introduction

Research Data Management (RDM) has evolved as a robust practice for data stewardship in Canada and internationally, and Canadian post-secondary institutions are currently developing strategies to build capacity in this realm to give assistance to their researchers. Strong RDM practices are encouraged to support researchers in achieving scientific rigour and to enable collaboration in their disciplines (Tri-Agency, 2021). Therefore, the question is: how do research institutions evaluate, compare, and improve on the suite of services they intend to offer to support sound research data management practices?

Why Canadian institutions would  want to conduct an RDM maturity assessment

In Canada, there are three major federal granting agencies: Social Sciences and Humanities Research Council of Canada (SSHRC); Natural Sciences and Engineering Research Council of Canada (NSERC); and Canadian Institutes of Health Research (CIHR). These agencies are known as the Tri-Agency and in 2021 they published their Research Data Management Policy. The objective of the policy states that it “is to support Canadian research excellence by promoting sound RDM and data stewardship practices” (Tri-Agency, 2021).

To follow the policy, the researcher needs the support of their institution; and to support their researchers, the institution needs to develop an overarching RDM strategy. Before this RDM strategy is developed, the RDM offerings at that institution should first be assessed. The results of this assessment will then inform the RDM strategy development. The best practice for the assessment involves first, to determine the assistance being offered during the different stages of the research data life-cycle. After that, it should be where the responsibility lies for the aforementioned supports. There are already some excellent international RDM maturity assessment models in existence (e.g., RISE, SPARC, ANDS), but every country has its own individual research requirements. Before 2021, there was no such RDM maturity assessment model for Canada. When institutions in Canada used the international models, some sections of the models were not applicable to the Canadian context and, likewise, other sections specific to Canadian institutions were not mentioned. Additionally, Canada is officially bilingual and there are few existing French RDM resources available to satisfy the requirements of a bilingual country.

After some research, it was decided to base the Canadian model primarily on the RISE framework, licensed under the Creative Commons Attribution 4.0 International licence, with elements also inspired by the SPARC tool and the ANDS framework. Given the above, a working group under the Digital Research Alliance of Canada (the Alliance) developed the RDM Maturity Assessment Model in Canada (MAMIC) to respond to the need for a Canadian-specific maturity assessment model available in both English and French. It should be noted here that  this maturity assessment model can be used by other international institutions as it is reflective of the evolving RDM landscape.

Completing the maturity assessment model will help the various RDM stakeholders at a research institution become more knowledgeable about the state of their RDM offerings. The process is also meant to be a collaboration among stakeholders who have a deeper understanding of these various RDM processes. These stakeholders include, but are not limited to, the Library; the Research Office; the Ethics Office; and the IT department. At the completion of the MAMIC, the stakeholders can use the information to review any gaps in RDM support and determine how these gaps will be addressed. 

Application of the MAMIC

Completing the MAMIC requires that information on various RDM resources and supports be delineated according to four categories: Institutional Policies and Procedures; IT Infrastructure; Support Services; and Financial Support. Within each category are elements specific to that topic and these elements are assessed on a 5-level scale according to their completeness or maturity level. These range from “Does not exist OR Do not know” to “Element is robust and focuses on continuous evaluation”. There is also an option to skip a certain element if it is not applicable to that particular institution. Another aspect of the evaluation is the assessment of the scale of each element, that is, a 3-level scale from “Offered only to specific users upon request” to the highest level of “Available to everyone”. This latter assessment also includes a “Not applicable” option. For example, when assessing the scale of support given to the creation of data management plans, an institution that limits support only to researchers applying to a large grant (due to capacity), would select a scale of 1, “Offered only to specific users upon request”.

     Another important piece of information included after each category of the MAMIC (mentioned above) is the name(s) and job title(s) of the individual(s) that filled in the information for that particular category. This affords other stakeholders the opportunity to contact that person if there are queries around the information that was filled in for a particular element.

The MAMIC was launched at a national workshop in October 2021, with participants from small to large research institutions. The participants were able to choose their language of choice for the workshop they would be attending, either  English or French. Afterwards, participants gave positive feedback about the workshop’s value, especially for French speaking and bilingual institutions; they mentioned that Canadian bilingual resources are not always readily available for them to assess their RDM offerings. Since its inaugural launch, the Alliance has recommended the use of the MAMIC to support institutions to help them fulfill a requirement of the Tri-Agency to create an RDM Institutional Strategy.

Completion of the MAMIC is intended to be for the sole use of the institution, to support the development of their RDM institutional strategies and service offerings. As such, the MAMIC is a standalone tool, the results of which are not designed to be shared with the Alliance or elsewhere.  The MAMIC development team has received positive anecdotal feedback from adopters of the model, and the team continues to monitor the release of published institutional strategies and how the tool has served those institutions.

The Carleton University Office for Research Initiatives and Services completed the MAMIC in September 2021. The RDM Institutional Strategy Working Group applied the recommendations given in the MAMIC when assembling the team that would complete it. The team included the Director and Assistant Director of the Carleton Office for Research Initiatives and Services; the Data Services Librarian; the Head of Research Computing Services; and the Director of Industry and Partnership Services. The team was not surprised by some of the results, as they had already realised that more institutional support was needed for RDM. However, it was very helpful to have a starting benchmark against which to measure the progress upon completion of  the MAMIC in future years. It was also very helpful to have evidenced-based information to present to the appropriate parties when asking for funding to fill the different gaps in institutional RDM support.

The University of Calgary also used the MAMIC. Their RDM Institutional Strategy (IS) Working Committee and Steering Committee were both co-led by the Office of the Vice President (Research), and Libraries and Cultural Resources. The Working Committee did the actual work completing the MAMIC, and then presented it to the Steering Committee for the go-ahead to move forward in the process of creating the RDM IS. This group realised that it was necessary to include other stakeholders in filling out the MAMIC, as the categories are too diverse for one individual to complete it.

The Memorial University of Newfoundland (MUN) also found the MAMIC to be quite helpful. They had a working group that was a sub-committee of the Vice President’s Council. Three  stakeholders, including the RDM Librarian, and staff members from the Research Initiatives and Services Office, and the Office of the Chief Information Officer, filled out the MAMIC, after which they presented their findings to the RDM IS working group. This group then used those findings to help develop their institutional strategy. As with the University of Calgary and Carleton University, MUN could not have fully completed the MAMIC without the aid of various campus stakeholders for the different service categories.

McMaster University used the RISE framework (Research Infrastructure Self Evaluation) to help determine the gaps in their RDM supports and services. This assessment model predated the MAMIC and was available when MacMaster was ready to assess their institutional RDM offerings. Once the MAMIC was created, they also chose to fill it in to gather more information, as it contained categories that were not in the RISE model and that were relevant to Canadian universities.

Each completion of the MAMIC at the various institutions looked very similar. They all had a small working group to complete it, composed of librarians, ITS representatives, research office members, and either a researcher or an industry partner. At another institution, the Data Librarian attempted to fill in the entire document, reaching out to colleagues in the office of research services to help fill in gaps. In each case, the results of the MAMIC were taken to a larger RDM committee for discussion.

Similar to McMaster, the University of Ottawa completed an assessment of the institutional RDM landscape prior to the release of the MAMIC. The MAMIC, however, served as a model to develop structured questions that were explored during focus groups conducted with the research community for feedback on a draft of the institutional strategy. Campus service providers, including research ethics, information management, and information technology, led focus groups related to their service areas, focusing on the elements of the MAMIC to guide the dialogue. The results of the focus groups were used to further develop the institutional strategy for different elements. This illustrates that the MAMIC can also serve as a valuable tool for institutions at different stages of the RDM IS development process.

A practical guide on how to implement the results of the MAMIC

The results of the MAMIC help institutions establish an understanding of the current state of RDM services, resources, staffing and external support. From there, it is important to reflect and determine the desired near-to-medium-term future state for RDM for the institution. That is, what needs or resources are required to advance from the current state to the desired state?

Here are some reflective questions to guide in the review of a specific section.

• Are there certain technical supports that are needed right away?
• Are there certain resources that are lacking, e.g., is there insufficient training for RDM on campus?
• Does the research community know who can give them IT support?
• Is the Research Office on board with helping to support researchers in the RDM area?

Suggestions for future and regular usage of the MAMIC

The RDM MAMIC is intended to be used for regular assessment of RDM services and supports. Institutions that prioritise research data management will remain competitive in an evolving research landscape, one that emphasises research excellence by ensuring that research is performed ethically, rigorously and collaboratively.

Some universities and colleges have already determined how often they will be completing the MAMIC again in the future.For example, McMaster University specifically states in their RDM Institutional Strategy that they will be evaluating their RDM services and supports annually using the MAMIC.

The original intent of the MAMIC is that it would have continued relevance, and, to this end, it will be re-evaluated in 2024 to update it, where needed. For example, there could be new IT elements that need to be added. As well, the MAMIC team will examine whether there are any differences between the English and French approach to assessing RDM services and supports. The MAMIC can also be adopted for use in other countries. The sections that comprise the MAMIC are service areas of RDM offerings across an institution, such as IT and HR. These are not unique to the Canadian context, and can be applied to other institutions.

References

ANDS. (2018, March 23). Creating a data management framework. https://web.archive.org/web/20220309174711/https://www.ands.org.au/__data/assets/pdf_file/0005/737276/Creatinga-data-management-framework.pdf

Fry, J., Abel, J. Dearborn, D., Farrell, A. & Ripp, C. (in press). The RDM Maturity Assessment Model in Canada (MAMIC). In K. Thompson, E. Hill, E. Carlisle-Johnston, D. Dennie, E. Fortin, (Eds.), Research data management in the Canadian context: A Guide for Practitioners and Learners. University of Western Ontario: Western Libraries.

Fry, J., Dearborn, D., Farrell, A., Khair, S., & Ripp, C. (2021). RDM Maturity Assessment Model in Canada (MAMIC) (1.0). Zenodo. https://doi.org/10.5281/zenodo.5745493

Fry, J., Dearborn, D., Farrell, A., Khair, S., & Ripp, C. (2021). Modèle d’évaluation de la maturité de la GDR au Canada (MEMAC) (1.0). Zenodo. https://doi.org/10.5281/zenodo.5745894

Government of Canada (2021-03-15). Tri-Agency Research Data Management Policy. https://science.gc.ca/site/science/en/interagency-research-funding/policies-and-guidelines/research-data-management/tri-agency-research-data-management-policy

Government of Canada. (2021-10-29). Tri-Agency Research Data Management Policy – Frequently Asked Questions. https://science.gc.ca/site/science/en/interagency-research-funding/policies-and-guidelines/research-data-management/tri-agency-research-data-management-policy-frequently-asked-questions

Rans, J and Whyte, A. (2017). “Using RISE, the Research Infrastructure Self-Evaluation Framework, v.1.1”. Edinburgh: Digital Curation Centre. Available online: www.dcc.ac.uk/guidance/how-guides

SPARC Europe. Evaluate your RDM offering. https://sparceurope.org/evaluate-your-rdm-offering/