RDA AI in Action: Global Workshops Report (February 2025)
Source: AI in Action — RDA Global Workshops Report, February 2025 Published: February 2025 (report finalised March 2025) Publisher: Research Data Alliance (RDA) Author: Shalini Kurapati, PhD
Report overview and context
This report documents the RDA’s “AI in Action – How Researchers Leverage AI” workshop series, held on 13 and 18 February 2025. The workshops were a direct follow-up to the RDA’s State of AI Pilot Workshops (September 2024), which identified key challenges in AI adoption: reproducibility, data governance, and ethical considerations. Where the September 2024 sessions catalogued the problems, these February sessions showcased practical solutions and real-world implementations.
Two sessions were held to accommodate global time zones: a Europe/Americas/Africa session (144 registrants, 65 live participants) and an Asia/Oceania session (122 registrants, 59 live participants). In total, 266 participants from 36 countries registered. Speakers were drawn from research software engineering, biomedical sciences, digital humanities, health technology, and research infrastructure, representing institutions in the Netherlands, Italy, Switzerland, India, and Australia.
The RDA is a 15,000-member global community spanning 150+ countries, focused on interoperability and data sharing standards. Its workshops reflect community-driven, practitioner-level perspectives rather than funder or policy perspectives — this distinguishes this source from the UNESCO survey or UKRI strategic framework documents already in the vault.
A methodological note: the executive summary section of this report was itself generated using Claude Sonnet 3.5 and ChatGPT-4o, reviewed and confirmed by workshop attendees. RDA explicitly documented this AI use and disclosed the privacy-preserving configurations applied. This is a useful reference point for how open-science communities are handling AI-generated outputs.
Finding 1: Reproducibility is the dominant operational challenge for AI in research
Across both sessions, reproducibility in AI and machine learning research emerged as the central preoccupation of practitioner researchers. Multiple speakers independently converged on the same diagnosis: that the research community lacks standardised reporting requirements for ML models, that peer reviewers often lack the ML expertise to evaluate them, and that published AI models frequently cannot be replicated due to missing datasets, undocumented computational environments, and incomplete methodological detail.
Gavin Farrell (University of Padova/ELIXIR) presented the DOME framework (Dataset, Optimisation, Model, Evaluation), originally published in Nature Methods (2021), as a structured response to this gap. DOME provides standardised reporting guidelines for ML experiments and underpins a curated DOME Registry with 117 active users and 214 recorded ML studies. Giulia de Luca (University of Bologna) provided an applied case study: her attempts to replicate published lung cancer AI models from the National Lung Screening Trial dataset were obstructed by missing data, undefined environments, and incomplete documentation. Her solution — containerisation via Docker/Singularity — is consistent with community best practice but underscores that reproducibility currently depends on individual researchers’ initiative rather than enforced standards.
Sheffield relevance: The reproducibility challenge maps directly onto the output-explainability and bias-risk-notes fields in the tool schema. Tools where these fields are unknown or low are the instruments most at risk of the exact failure mode this session describes. Sheffield researchers publishing AI-assisted findings face the same accountability gap the RDA community is working to close. The DOME framework is a candidate for addition to the tags vocabulary or as a reference in the schema documentation for fields related to ML model governance.
Finding 2: Generative AI is entering research software engineering — with human oversight as a non-negotiable requirement
Carlos Utrilla Guerrero (TU Delft) presented the institution’s GenAI4RS training initiative, which embeds hands-on LLM workshops into research software engineering practice. The practical takeaway was not that GenAI tools are unreliable, but that their reliability depends on validation practices: robust versioning of AI models, human-in-the-loop oversight, adherence to FAIR4RS (FAIR for Research Software) principles, and clear audit trails for AI-generated research artefacts.
Brian Ballsun-Stanton (Macquarie University) demonstrated a more advanced use case in digital humanities: structured LLM prompting to synthesise 160 academic sources into an organised bibliography with narrative explanation (“Calculator for Words”). He also demonstrated using the Claude API for academic copy-editing with transparent change-tagging. His key policy positions were that AI-generated outputs should be disclosed at a standard comparable to software citation, and that LLM transcripts should be published openly (his team does this via OSF). These are practitioner-level implementations of standards the RDA and broader research community have not yet codified.
Sheffield relevance: The TU Delft and Macquarie examples represent what mature, governed use of generative AI in research workflows looks like. Both institutions have moved beyond ad hoc use to structured practice with explicit oversight. Sheffield’s position — GenAI Principles adopted, operational catalogue under development — places it ahead of the governance stage but behind the maturity stage these institutions demonstrate. Tools assessed in the coding-and-development and scholarly-writing capability areas, particularly those using LLM APIs directly, should be evaluated against the FAIR4RS criteria that TU Delft applies. The human-oversight-required field in the schema is validated by both presentations as a non-negotiable operating condition, not a precautionary annotation.
Finding 3: Domain-specific AI in health and research classification presents distinct governance challenges
Two presentations addressed domain-specific AI in high-stakes contexts. Susannah Soon (Curtin University) presented Healthy Connections, a portable AI-enabled Medi-Kit combining diagnostic IoT devices, a multilingual AI virtual health assistant, ECG analysis via LLMs, and computer vision for skin condition detection. The system is designed for deployment in Indigenous and rural communities with limited healthcare access. Key governance concerns include data sovereignty, CARE principles (Collective Benefit, Authority to Control, Responsibility, Ethics) alongside FAIR, and model reliability for populations that may be underrepresented in training data.
Amir Aryani (Swinburne University) presented Research Link Australia, a national research classification platform using a ~22 billion parameter LLM for zero-shot Fields of Research classification, with human-in-the-loop validation by subject matter experts. The governance challenges highlighted were different in character: identity disambiguation, bias in automated classification, data integration across persistent identifier systems, and transparency in how AI assigns classifications that subsequently shape funding decisions.
Sheffield relevance: Neither of these specific tools is a direct candidate for the Sheffield catalogue in its current scope (research infrastructure rather than researcher-facing tools). However, the governance dimensions they expose are relevant. The CARE principles — which the RDA session foregrounded as a complement to FAIR specifically for Indigenous data contexts — are not currently represented in the tool schema. For tools used in research involving communities with data sovereignty concerns, this is a gap. As with the international-partnership-risk tag proposed in response to the UNESCO survey, this does not require a schema change: a care-principles-applicable tag could be used in the tags field for relevant tools. The health AI discussion also reinforces that bias-risk-notes must be populated with domain-specific content, not generic disclaimers, before any tool in a sensitive research domain advances to catalogue-status: adopt.
Finding 4: Community standards are emerging but not yet enforced — the gap between policy and practice is the central risk
The discussion sessions across both workshops returned repeatedly to one structural problem: the gap between the existence of standards (FAIR, DOME, FAIR4RS, CARE) and their actual enforcement in journals, funding bodies, and institutional practice. Participants noted that most journals lack AI/ML reporting guidelines, that reproducibility depends on researchers’ voluntary choices rather than requirements, and that model drift and versioning are not systematically addressed.
The workshop discussions also surfaced the FAIRlyze platform (from RDA’s AIDV Working Group), which offers automated quality control on datasets using OpenAI tools, as an emerging instrument for operationalising governance standards. The session discussions explicitly raised the need for AI model audits, documentation of third-party API dependencies, and environment snapshots as baseline reproducibility infrastructure.
Post-session feedback (12 respondents from Session 1, 6 from Session 2) indicated strong appetite for deeper engagement: participants requested more focused treatment of AI limitations (bias, outdated data), practical guidance on system architecture, and greater attention to AI in humanities and social sciences — a field the RDA workshop organisers themselves noted is underrepresented.
Sheffield relevance: The policy-practice gap the RDA community diagnoses is precisely the gap the current project addresses at institutional level. Sheffield has the policy (GenAI Principles); the catalogue is the operationalisation layer. This report’s findings validate the catalogue’s governance-first approach. The note on humanities and social sciences underrepresentation is directly relevant: of the 14 capability areas in the Sheffield landscape, qualitative-coding and scholarly-writing are the areas with the most direct humanities application, and both are less thoroughly assessed than STEM-oriented areas. The RDA finding supports prioritising humanities tool coverage in a future sprint.
Implications for this project
| RDA finding | Catalogue implication |
|---|---|
| Reproducibility is the dominant practitioner challenge | output-explainability and bias-risk-notes are load-bearing governance fields, not optional annotations |
| DOME framework provides a standardised ML reporting structure | Consider referencing DOME as a governance standard in schema documentation; dome-compliant could be added as a tag for ML-specific tools |
| Human-in-the-loop oversight is the practitioner consensus, not a theoretical requirement | human-oversight-required: true is already correct as a schema default; emphasis in tool notes should move to how oversight is implemented, not just that it is required |
| GenAI in RSE: audit trails and FAIR4RS compliance are the emerging standard | For tools in coding-and-development, FAIR4RS compliance and AI output versioning should be added to the trust and governance notes section |
| CARE principles alongside FAIR for community/health data contexts | care-principles-applicable tag (using existing tags field) for tools used in research with Indigenous or community data sovereignty considerations |
| Humanities and social sciences AI use is underrepresented globally | Prioritise qualitative-coding and digital humanities tool coverage in next sprint assessment |
| Policy-practice gap is the sector-wide risk | The catalogue’s role is precisely to close this gap at Sheffield — validates the project’s strategic positioning |
Recommended adjustments to the catalogue
Three proposals are suggested by these findings. None require immediate action in this commit — they are documented here as a record for future sprint planning.
1. Reference DOME in schema documentation
The DOME framework (Walsh et al., Nature Methods, 2021; DOI: 10.1038/s41592-021-01205-4) provides a validated standard for ML model reporting that maps onto existing schema fields. Adding a reference to DOME in the _schemas/tool-note-schema.md documentation for output-explainability and bias-risk-notes would give assessors a concrete checklist when populating those fields for ML-specific tools.
2. CARE principles tag for sensitive community data contexts
Use the existing tags free-text field to annotate tools as care-principles-applicable where the tool is likely to be used in research involving Indigenous, community, or other data-sovereign populations. This complements the international-partnership-risk tag proposed in response to the UNESCO survey and avoids a schema version increment.
3. Humanities sprint coverage
The RDA workshops explicitly noted underrepresentation of humanities and social sciences AI use. Of Sheffield’s 14 capability areas, qualitative-coding and scholarly-writing have the thinnest tool coverage relative to their potential user base. A targeted sprint assessing AI tools specifically for arts, humanities, and social science research workflows would address both the RDA-identified gap and a real Sheffield researcher need.
Drafted: 2026-04-01 · Branch: agent/2026-04-01-rda-ai-in-action-survey