Embodied First Exam Introduces New Robotics Evaluation Framework

The Embodied First Exam, a new robotics evaluation framework, has been introduced to assess the capabilities of coding agents in controlling robots under physical-world constraints. Published on November 9, 2026, on arXiv (2611.12177) and highlighted by HuggingFace Daily Papers, the framework is designed by a team led by Kangrui Wang, Qineng Wang, Changhe Chen, Pingyue Zhang, and Chi Gui. It aims to address gaps in existing robotics benchmarks, which primarily focus on visuomotor policy learning, by allowing agents to choose their control approach: direct control, code to control, or training a policy to control.

Comprehensive Benchmark Features

The Embodied First Exam includes 1,156 tasks across 35 suites and 47 robots, with nine original suites specifically developed for this framework. These suites extend beyond traditional pick-and-place tasks to encompass spatial puzzles, dexterous manipulation, multi-robot interaction, dynamic whole-body control, and sim-to-sim engineering challenges. Notable suites include:

• RoboPaint: Tests fine tool control, such as reproducing images with a brush or pen.
• RoboARC: Focuses on manipulating deformable and multi-part objects, including tasks like threading ropes and assembling furniture.
• DexHandExam: Evaluates dexterous hand manipulation, such as turning book pages and opening containers.
• Embodied Olympics: Features humanoid robots performing sports-inspired tasks and competitive matches against opponents.

Each task can be executed in three evaluation settings: Standard, Realtime, and Privileged. The Standard Setting mimics real-world conditions, providing only sensor observations and limiting environment resets. The Realtime variant keeps the environment running continuously, while the Privileged Setting grants full simulator access and unlimited resets, enabling agents to explore optimal solutions for training purposes.

Performance and Open-Source Availability

In testing, the strongest of eight agents, GPT-6 Astra in Codex, achieved a 34% success rate on 456 core tasks in the Standard Setting, highlighting significant room for improvement. The framework also includes tools to analyze failure points across agents, settings, and task stages.

The Embodied First Exam is open-source and available on GitHub at https://github.com/mll-lab-nu/embodied-first-exam. It requires Python 3.12 or later and supports GPU-accelerated tasks via the NVIDIA Container Toolkit. The framework can be installed using the Harbor package manager, with detailed instructions provided in the repository.

This benchmark represents a significant step toward advancing agentic robotics, offering a unified platform for evaluating coding agents before their deployment in real-world robotic applications.