Leveraging Task-Specific Large Language Models to Enhance Research Data Management Services

February 10th, 2025

Introduction

Expectations for researchers to manage and openly share their data have grown substantially over the past decade, driven by efforts to improve research reproducibility and expand opportunities for data reuse. The 2022 OSTP Nelson Memo: Ensuring Free, Immediate, and Equitable Access to Federally Funded Research is set to continue and advance this trend as it directs all US federal funding agencies with research and development funds to establish or update public access policies that include data management and sharing requirements (Nelson 2022). A key outcome of this policy shift will be the widespread adoption of data management and sharing plans (DMSPs) as mandatory components of federal funding proposals. 

This increased focus on data management and sharing necessitates that researchers develop new skills while institutions must adapt their service models to support researchers and compliance effectively. To meet these demands at scale, institutions must explore innovative and sustainable solutions. One potentially promising approach is leveraging generative AI, specifically Large Language Models (LLMs), in combination with prompt engineering (Liu et al. 2023; Chen et al. 2024) and retrieval-augmented generation (RAG) (Lewis et al. 2020). This approach enables us to tailor LLMs to perform task-specific functions. Together these approaches could effectively create opportunities to optimize workflows, revolutionize how we support researchers, reduce researcher burden, and support adherence with evolving policies.

Overview of the Approach and Technologies

Pre-trained LLMs are powerful and versatile tools that can partake in a wide range of tasks (Bubeck et al. 2023). However, they also have fundamental limitations, such as a lack of domain-specific knowledge, inaccurate content or hallucinations, and an inability to provide up-to-date information. By applying prompt engineering and RAG techniques, institutions can tailor generative AI capabilities to address specific needs efficiently, without the extensive cost and time associated with retraining custom models. These approaches address many of the inherent limitations of pre-trained LLMs while also enhancing their real-world applicability. 

Prompt Engineering: What is it and what issues does it address?

Prompt engineering involves crafting structured inputs that guide a LLM to generate more relevant and context-specific outputs. By providing detailed instructions, contextual frameworks, and/or examples (e.g., user interactions or demonstrations), institutions can focus LLMs on achieving specific objectives (Nori et al. 2023; Sahoo et al. 2024). 

Several prompt engineering techniques can help address common issues with LLMs:

• Few-shot prompting: This technique involves providing a model with a few high-quality examples of potential input-output scenarios. The benefit of this technique is that it can enable a LLM to partake in new tasks without extensive retraining, thereby improving domain specificity and accuracy (Nori et al. 2023; Sahoo et al. 2024). 
• Chain-of-Thought Prompting: This technique encourages the model to break down complex problems into smaller, sequential steps. The benefit of this technique is that it increases the likelihood of relevant and accurate responses, therefore reducing hallucinations (Nori et al. 2023). 

Retrieval-Augmented Generation (RAG): What is it and what issues does it address?

RAG enhances the capabilities of LLMs by incorporating them with an external, curated database containing authoritative, up-to-date information. When a query is received, the model retrieves relevant, domain-specific content from the curated database before generating a response. By grounding the LLM’s output in a trusted source of information, RAG addresses several limitations of LLMs, including domain specificity, hallucinations, and timeliness of information. RAG ensures outputs are more accurate and contextually relevant because responses are informed by a reliable, curated database and not solely based on the LLM’s pre-trained data. Additionally, the ability to update the curated database ensures that the information used remains current. 

Therefore, an approach which adopts prompt engineering and RAG to customize pre-trained LLMs for task-specific functions can offer several benefits including customization without extensive retraining and enhanced accuracy of generated responses.  

Potential Applications in Data Management and Sharing Services

Applying prompt engineering and RAG to research data management and sharing activities offers numerous opportunities for enhancing institutional research data support services. Here, we present just a few illustrative examples that highlight how these technologies could significantly improve service efficiencies, reduce researcher burden, and support adherence with evolving policies. These examples aim to inspire further exploration and future work rather than serve as extensive case studies.

• Task-Specific, Agent-Based Chatbots for Data Management and Sharing Plans (DMSPs): Agent-based chatbots can assist researchers in drafting DMSPs by prompting for specific information based on funder requirements. This would offer researchers an interactive, guided experience that streamlines the process of developing a DMSP. The chatbot can be pre-loaded with knowledge of DMSP policies, institutional resources, and common pitfalls observed during plan reviews. Moreover, by incorporating review criteria, these chatbots could also provide real-time feedback on draft plans, allowing researchers to refine their submissions before institutional review. 

• Automated Text Extraction for Structured Compliance Reporting: Using these approaches, institutions can also automate the extraction of key details from narrative-based DMSPs and transform them into structured, formatted fields. This could be particularly useful for converting narrative-based DMSPs into actionable steps for researchers, service providers, and compliance officers, enabling efficient monitoring and follow-up on data management and sharing commitments. 

• Customized Knowledge Retrieval for Policy Guidance and Updates: Institutions can further leverage these approaches to develop tools that offer researchers up-to-date guidance on data management and sharing policies from major funders and publishers as well as institutional requirements. For instance, a researcher could query these tools to receive the latest mandates, institutional requirements, or best practices related to data management and sharing. This capability would reduce the burden for researchers in tracking down the most recent policy update.

Challenges and Limitations

Despite the promising benefits of using prompt engineering and RAG to scale data management and sharing services, several challenges and limitations remain. While these approaches can improve the accuracy and reliability of responses, their effectiveness depends heavily on factors such as prompt quality and the accuracy of the curated database used for retrieval. If prompts are poorly designed or the curated database is outdated, the generated outputs may still contain inaccurate information. Suboptimal prompts can lead to incorrect or misleading outputs and the quality of the information retrieved through RAG is only as good as the curated database. Ensuring the database remains up-to-date and comprehensive is a continuous challenge that will need to be met. 

Additionally, integrating LLMs into data management and sharing workflows raises concerns around data privacy and security. Researchers may need to share sensitive information during the proposal stage, including unpublished data. This necessitates restricting access to the researcher provided input. One solution is to deploy an on-premise LLM or institutionally controlled systems. 

Further, implementing and maintaining AI-driven solutions will require both technical infrastructure and ongoing support. Institutions will need to invest in resources for maintaining these systems as well as technical and user-focused personnel to support update and use of these systems.

References

Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., … & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712. https://doi.org/10.48550/arXiv.2303.12712

Chen, B., Zhang, Z., Langrené, N., & Zhu, S. (2023). Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review. arXiv preprint arXiv:2310.14735. https://doi.org/10.48550/arXiv.2310.14735

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459-9474. https://dl.acm.org/doi/abs/10.5555/3495724.3496517

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1-35. https://doi.org/10.1145/3560815

Nelson, A. (2022). Memorandum for the heads of executive departments and agencies: Ensuring free, immediate, and equitable access to federally funded research. Office of Science and Technology Policy. https://bidenwhitehouse.archives.gov/wp-content/uploads/2022/08/08-2022-OSTP-Public-Access-Memo.pdf

Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., … & Horvitz, E. (2023). Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 https://doi.org/10.48550/arXiv.2311.16452.

Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., & Chadha, A. (2024). A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927. https://doi.org/10.48550/arXiv.2402.07927