Paper: Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

Papers
Page content

Listen to this article.

Only the latest audio is kept; older files are removed on each update.

Problem

Autonomous AI agents are showing promise in conducting machine learning research. However, these agents often lack “operational knowledge”—the practical, domain-specific know-how that bridges the gap between understanding a technique and applying it effectively. This knowledge currently resides in the vast landscape of code repositories and research papers, but it’s scattered, difficult to access, and too large to efficiently use during task execution.

Method

The paper introduces DisCo, a skill-powered research agent designed to address this issue. DisCo employs a “distillation” process to convert existing code repositories into reusable “skills.” This distillation happens in two ways:

  • Task-agnostic distillation: It analyzes widely used ML repositories (1,000 in this case) to create a library of general-purpose skills (the AREX-Skill Library, containing over 5,000 skills).
  • Task-oriented distillation: It produces skills specifically tailored to the requirements of a particular research task.

These skills are then integrated into an agent with a GPT-5.5 backbone and research harness.

Results & Limitation

According to the authors, equipping the research agent with these skills significantly improves its performance on several benchmarks: MLE-bench, PaperBench, FrontierCS, and PassNet. Specifically, they report an average performance increase of over 130% on MLE-bench and improvements across the other benchmarks.

It’s important to note that this assessment is based solely on the abstract. It’s uncertain whether these improvements are consistently reproducible, what the computational cost of skill distillation and integration is, and the robustness of the AREX-Skill Library to changes in the ML ecosystem. The abstract doesn’t provide details on the verification process for the distilled skills.

Why It Matters

This work could have significant implications for the development of autonomous ML research agents. By enabling agents to leverage pre-distilled knowledge from the ML community, it might accelerate the pace of research and reduce the need for agents to “reinvent the wheel.” For data scientists and ML practitioners, this suggests a future where readily available, reusable skills could be integrated into automated workflows, improving efficiency and potentially uncovering new insights. The creation of the AREX-Skill Library is particularly exciting as it provides a valuable resource for the broader ML community.

References