Paper: Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Den...

Papers
Page content

Listen to this article.

Audio is available for 30 days and will be removed automatically.

Problem

Current benchmarks used to evaluate AI agents often focus on simple tasks that complete quickly and are judged solely by their final outcome. This doesn’t give a full picture of an agent’s capabilities, especially when dealing with complex, real-world scenarios requiring sustained effort and iterative problem-solving. Existing “terminal” benchmarks (which judge only the end result) provide limited insight into intermediate progress and partial solutions due to sparse reward signals.

Method

The paper introduces “Long-Horizon-Terminal-Bench,” a new benchmark designed to address this issue. It consists of 46 long-horizon tasks spread across nine categories like experiment reproduction, software engineering, and scientific computing. Crucially, each task is broken down into smaller, graded subtasks that provide dense intermediate rewards. This allows the evaluation system to assess progress beyond just reaching the final goal. The tasks are designed to take hundreds of episodes and minutes or hours to complete, emphasizing skills like long-horizon planning, managing large contexts, and iterative debugging. They utilize a “Terminal-Bench” style setup with reference solutions or simulation engines for verification.

Results & Limitation

The authors tested 15 state-of-the-art models on Long-Horizon-Terminal-Bench, revealing that even the best model only achieved 15.2% success rate. These agents consumed a substantial amount of resources: approximately 9.9 million tokens per task, requiring roughly 231 episodes and 85 minutes of execution time on average. A limitation from just looking at the abstract is we don’t know what metrics were used beyond the final success/failure percentage; more granular performance details would be helpful for understanding where agents struggled.

Why It Matters

This benchmark offers a significant step forward in evaluating AI agent capabilities, moving beyond simple “end result” judgements to assess progress and problem-solving approaches across extended timelines. Data scientists and ML practitioners involved in building autonomous agents—particularly those dealing with complex workflows or scientific tasks—will find Long-Horizon-Terminal-Bench valuable for benchmarking and identifying areas for improvement. The shift towards dense rewards and assessing intermediate steps is a crucial advance toward more robust agent evaluations.

References