Paper: Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning

Papers
Page content

Listen to this article.

Only the latest audio is kept; older files are removed on each update.

Problem

Current vision-language models excel at recognizing and describing physical events, but they struggle with the deeper task of reasoning about how those events unfold and how to predictably influence them. This boils down to a lack of explicit representation of the underlying physics – things like object states, how objects interact (dynamics), and measurable parameters governing their behavior. Essentially, these models understand what happens but not necessarily why.

Method

The authors introduce “Code-as-World,” a novel approach that tackles this issue by representing physical worlds as executable code. This means the “world” isn’t just an image or text description; it’s a set of instructions that can be run to simulate its behavior. They use an “agentic discovery loop” inspired by abductive reasoning. An agent iteratively:

  1. Proposes a potential executable world hypothesis (i.e., code).
  2. Executes the code (simulates the world).
  3. Renders a visual output based on the execution.
  4. Verifies if the rendered output aligns with observations (e.g., real-world videos, text descriptions).
  5. Refines the executable world hypothesis based on this verification process.

Results & Limitation

The authors claim that using these “verified executable worlds” as a form of scalable physical supervision allows them to train vision-language models (“Code-as-World-VL”) that achieve state-of-the-art performance on tasks like QuantiPhy (a benchmark for quantitative physical reasoning). They even surpass the performance of some proprietary models.

It’s difficult to gauge the true limitations from just the abstract. We don’t know how scalable this “agentic discovery loop” is in practice—generating and verifying these executable worlds could be computationally expensive, especially for very complex scenarios. Additionally, the robustness of these representations to noisy or incomplete observations remains an open question.

Why It Matters

For data scientists and ML practitioners working on robotics, autonomous navigation, simulation-based training, or any application requiring a deep understanding of physical interactions, this work is potentially groundbreaking. The idea of representing the world as executable code provides a quantitatively grounded way to build models that can reason about physical systems. If “Code-as-World” proves scalable and robust beyond the benchmarks presented, it could represent a significant advance towards achieving more general and capable artificial intelligence.

References