Research AI Landscape

sakana-ai-scientist

3 min read

AI Scientist (Sakana AI)

Status: assess | Maturity: experimental | Last reviewed: 2026-04-15

What it does

AI Scientist is an open-source agentic system from Sakana AI designed to automate the full machine learning research cycle: generating research ideas, writing and executing code, running experiments, analysing results, and writing a paper. Version 2 (AI Scientist-v2, Yamada et al., 2025) uses agentic tree search to generate and refine code implementations without relying on human-coded templates.

In 2025, AI Scientist-v2 produced the first fully AI-generated paper to be accepted at a peer-reviewed workshop — specifically, an ICLR workshop. This is a notable milestone, though the paper underwent automated rather than full expert review, and the milestone marks the system’s capability rather than its scientific reliability.

Key capabilities

  • End-to-end research pipeline from idea generation to manuscript writing
  • Agentic tree search for code generation and refinement (v2)
  • Automated experiment execution and results analysis
  • LaTeX paper writing including figures and references
  • Open-source — code available on GitHub

Fit for Sheffield researchers

Currently of interest primarily as a research subject rather than a research tool. The system demonstrates what agentic AI can produce autonomously, which is directly relevant to discussions about AI authorship, research integrity, and the future of automated science.

Practical research use at Sheffield is not recommended at this stage: the system requires significant LLM API spend per run, produces outputs that require expert validation, and is optimised for ML research (model training and evaluation) rather than broader scientific domains. It is not a general-purpose research assistant.

Monitor for v3 and subsequent releases — the trajectory is fast.

Licensing and access

Open-source code available on GitHub under a permissive licence. Running the system requires a paid LLM API backend (typically OpenAI or Anthropic APIs). API costs per full research run are substantial and depend on the LLM backend chosen. No Sheffield licence or procurement route exists.

Trust and governance notes

The ICLR workshop acceptance was notable but should not be over-interpreted: workshop papers receive lighter peer review than main conference papers, and automated review processes were used. Human expert scrutiny of AI Scientist outputs has identified cases of superficially plausible but scientifically flawed work.

Key research integrity concern: the system optimises for producing outputs that appear publishable rather than outputs that are scientifically correct. Sheffield researchers using this system should treat its outputs as draft starting points requiring rigorous expert review, not as research-ready contributions.

No data handling or GDPR documentation is available for the system itself; the underlying LLM API providers’ terms apply to any data sent during runs.

Observed limitations

  • Validated primarily on ML research tasks; untested across most scientific domains
  • Accepted paper was at a workshop (lighter review), not a main conference or journal
  • Outputs can be superficially plausible but scientifically flawed
  • Requires expert oversight to distinguish genuine discovery from artefact
  • High operational cost per run; not suitable for routine use
  • Community is small and independent benchmarking is limited

google-ai-co-scientist · aletheia · chemcrow

Sources and evidence

Graph View

Backlinks