01Agentic Robotics Benchmark

Agents’ Last Examfor Robotics

Each task gives an AI agent one stage of a robot engineer’s job, or every stage in order. A grader the agent never sees scores what it leaves behind.

Get started

Fig. 1Images and videos from the benchmark’s tasks. Pick a plate to open its platform.

01Requirements
02CAD
03Assembly
04Description
05Control
06Environment
07Behavior

02Robot platforms

The seven stages, platform by platform

Each platform fills the same seven stages with its own kind of work, and skips any stage it does not have. On a quadruped, control means a gait controller; on a car, the robot description is a vehicle model.

Robotic arm

Example task chain

The chain starts with a customer brief for a six-servo desktop arm and ends with a fetch in a hidden workcell. Each task has its own hidden unit or cell, and the agent has to work out the numbers from photos, datasheets and a test bench, as an engineer would.

01–03 are benchmark pull requests under review (#13, #15, #16); the rest are in development.

Design

01 · Requirements

Replica requirements from photos

The agent reads bench photos, a build log and datasheets, then writes a traced requirements sheet, a bill of materials and a kinematic sketch.

02 · Hardware design

CAD design from photos

The agent models the ten printed parts and their assembly in FreeCAD, working from 80 uncalibrated photos and the servo's STEP file.

Build

03 · Assembly

Full assembly by a Franka Panda

The agent writes the program that has the Panda build all 15 mates of the arm from loose parts, guided by a 25-photo picture manual and cameras.

04 · Robot description

MJCF from photos and servo readings

The agent calibrates a MuJoCo model of the photographed arm from hand-held photos and the joint readings logged with them.

Execute

05 · Control

Commission two customer units

Starting from a nominal model, the agent spends a budget of experiments on a test bench and delivers a calibrated model and a controller for each unit.

06 · Environment

Workcell twin from a phone survey

The agent builds a MuJoCo digital twin of a drawer workcell from ten survey photos taken with unknown cameras.

07 · Behavior

Fetch with cameras only

The actor fetches from the table and from closed drawers with two RGB-D cameras and encoders whose zero offsets nobody recorded.

End to end · 01 → 07

08 · End to end

Replicate a lost arm, then fetch with it

One agent does all seven stages. The grader builds the arm from that agent's CAD, then runs its assembly and commissioning programs and actor in the hidden cell.

Quadruped

Workflow map

Control on a quadruped is leg IK plus a gait scheduler or MPC, and tracking a commanded body velocity counts as control even when a learned policy does it. The task world would be an obstacle course.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Motor and gear-ratio sizing

Given a brief and a component catalog, the agent would size a motor and gear ratio for every joint, pick a battery and write the bill of materials.

02 · Hardware design

Legs and body frame around the actuator

Printable leg linkages, a body frame, motor mounts and an electronics bay would all be designed around the actuator module's STEP.

Build

03 · Assembly

Leg modules mounted onto the body

The deliverable would be a program that makes an assembler robot build the quadruped in simulation, using the printed parts, the actuators and the mating graph.

04 · Robot description

Floating-base MJCF with foot contacts

The model, built from the assembly and its meshes, would have a floating base, foot contacts, PD actuators and a standing keyframe.

Execute

05 · Control

Leg IK and a gait controller

Leg IK and a gait scheduler or MPC, written against the model, would track commanded body velocities without falling.

06 · Environment

Obstacle course with gates and terrain

Text and goal photos would become a course of gates, stairs and gaps, with a Gym wrapper, a success check and an oracle run.

07 · Behavior

Course runs and fetch missions

The actor on the gait controller would have two missions: pass the gates in order, or find an object, reach it and return.

End to end · 01 → 07

08 · End to end

Design a quadruped, then run the course

One agent would design the quadruped and its stack from an actuator's CAD, a brief and course photos. The score would be the gates it passes in order on hidden courses.

Autonomous driving

Workflow map

Design starts from a vehicle parameter sheet, and the only CAD is a deck and mounts on a bought chassis. The robot description is a single-track vehicle model with tire terms, and control begins by identifying its values from driving logs.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Vehicle parameter sheet and sensor suite

Using a racing brief and a component catalog, the agent would set the car's mass, wheelbase, steering and speed limits, choose its sensors and cost the BOM.

02 · Hardware design

Deck and sensor mounts

The agent would lay out a laser-cut deck and mounts for the LiDAR and computer on the chassis standoff pattern, around the component outlines.

Build

03 · Assembly

Deck and mounts fitted to the chassis

The program would drive an assembler robot in simulation, fitting the deck and the mounts onto the bought chassis.

04 · Robot description

Single-track parameters with tire terms

Two files describe the car: a single-track parameter file with Pacejka tire terms and a URDF with sensor frames, both taken from the build and component facts.

Execute

05 · Control

Tire identification and a path tracker

The agent would first identify the single-track and tire parameters from driving logs, then write a pure-pursuit or MPC tracker that holds a racing line.

06 · Environment

Race track with a Gym wrapper

The track would be an occupancy map and centerline drawn from a track brief or a top-down image, with a success check and an oracle lap.

07 · Behavior

Fast laps on unseen tracks

With only LiDAR and the car's pose to go on, the actor would lap unseen tracks quickly and avoid static obstacles and the walls.

End to end · 01 → 07

08 · End to end

Build a race car, then race unseen tracks

Starting from a racing brief, a parts catalog and the chassis outline, one agent would build the car and its race stack. The grader would time its laps on hidden tracks with obstacles.

Humanoid

Workflow map

Rod-driven ankles close kinematic loops, and the robot description has to keep them. Control is whole-body and can be an IK task stack, a QP balance controller, MPC or a learned walking policy.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Joint motors sized from motion clips

The agent would work out a DoF layout, a motor and gear ratio per joint, and thermal margins from a brief, reference motions and an actuator catalog.

02 · Hardware design

Legs, parallel ankles and a torso frame

Leg and arm mechanisms, a rod-driven parallel ankle and a torso frame would be designed around the actuator CAD so that they move without collisions.

Build

03 · Assembly

Ankle linkages and torso frame assembled

An assembler robot would build the legs and torso in simulation from the printed links, rods and actuators, driven by the agent's program.

04 · Robot description

Closed-chain model with IMU and contacts

Converting the assembly to a model means keeping the ankle's closed loop and adding an IMU, foot contact sensors and collision exclusions.

Execute

05 · Control

Whole-body IK and balance control

Under pushes, a whole-body IK stack and a balance controller on the model would keep tracking hand, foot and center-of-mass targets.

06 · Environment

Loco-manipulation scene or stepping course

The scene would be a room of tables, boxes and shelves or a stepping course, built from text or goal photos with its success check and an oracle.

07 · Behavior

Box carrying and course walking

Running on the whole-body controller, an actor would carry a box to a shelf or walk a stepping course to its goal.

End to end · 01 → 07

08 · End to end

Design a humanoid, then walk a stepping course

One agent would build a humanoid, its controller and a course from actuator CAD, a brief and course photos. The humanoid would have to reach the goal on hidden courses.

Dexterous hand

Workflow map

A hand has many small joints, often coupled through tendons, and the model must carry that coupling. Control adds retargeting, which maps human hand motion onto the robot hand.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

DoF layout and finger actuator sizing

The pinch forces an object set needs would set the DoF layout and drive type, and the servos chosen from a catalog would go into a BOM.

02 · Hardware design

Printable fingers, palm and tendon routing

Around a servo's stand-in CAD, the agent would design printable phalanges, a palm with thumb placement, and tendon channels and spools.

Build

03 · Assembly

Joints pressed and tendons threaded

Given the printed links, servos and spools, a program would make an assembler robot press the joints and thread the tendons.

04 · Robot description

Tendon-coupled MJCF with fingertip sites

The hand assembly would become a model whose coupled joints move through tendons or equality constraints, with a site at each fingertip.

Execute

05 · Control

Fingertip IK and human-hand retargeting

Control would pair a retargeting module, which maps human hand landmarks to joint targets, with IK on the model that solves several fingertip targets at once.

06 · Environment

In-hand scene with randomized objects

Starting from a task text, the agent would place the hand, randomize the mass and friction of an object set, and add a success check and an oracle.

07 · Behavior

In-hand cube rotation

On a floating wrist, the actor would rotate a cube in the hand to a target angle without dropping it.

End to end · 01 → 07

08 · End to end

Design a hand, then rotate a cube in it

Given servo CAD and a goal, one agent would make a printable hand with its model, IK, scene and actor. The grader would check the rotation on hidden cubes.

UAV

Workflow map

A UAV's IK is thrust allocation plus a trajectory tracker, and its robot description is a small parameter set: mass, inertia, rotor poses, and thrust and torque coefficients.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Propulsion and battery sizing

The agent would choose a motor, propeller, ESC and battery from a catalog to fit the mission brief, and predict hover time and thrust-to-weight.

02 · Hardware design

Rotor layout and printable frame

The arms have to be long enough to clear the propellers; the rotor layout and printable frame would be drawn around motor, propeller and battery stand-ins.

Build

03 · Assembly

Motors and rotors mounted on the frame

In simulation, an assembler robot would put the airframe together from the frame plates, motors and propellers, run by the agent's program.

04 · Robot description

Rotor parameter set and a flyable model

Mass, inertia, rotor poses and thrust coefficients would come from the frame CAD and propeller data, written as MJCF or a PX4 airframe.

Execute

05 · Control

Thrust allocation and a trajectory tracker

A mixer turns force and torque into rotor commands, and a tracker built on the parameters holds setpoints in wind.

06 · Environment

Arena with gates, trees and wind

A gate track or a forest with a wind field would be built from a scenario spec and goal renders, plus a Gym wrapper, success check and oracle.

07 · Behavior

Gate racing and arena navigation

Using its own tracker, the actor would fly the gates in order, or cross a cluttered arena to a goal without crashing.

End to end · 01 → 07

08 · End to end

Build a quadrotor, then fly unseen arenas

One agent would build the quadrotor, its controller and an arena from a mission brief and a parts catalog. Its actor would then have to reach the goal in hidden arenas.

Space robot

Workflow map

Rovers, free-flyers, CubeSats and landers share the chain. In orbit, control is attitude control with reaction-wheel and thruster allocation, and the description can be a Basilisk spacecraft object.

05 Control is a benchmark task; the other stages have none yet and show what they would produce.

Design

01 · Requirements

Mass, power, link and delta-v budgets

Working from a mission brief and transcribed datasheets, the agent would close the mass, power, link, thermal and delta-v budgets and write a BOM.

02 · Hardware design

CubeSat, rover or free-flyer structure

A CubeSat frame, a rover chassis or a free-flyer bus would be designed around the component CAD and checked for fit, mass and first natural frequency.

Build

03 · Assembly

CubeSat stack or rover chassis assembled

The agent's program would drive an assembler robot that stacks the CubeSat or builds the rover from the frame parts and boards, in simulation.

04 · Robot description

Basilisk spacecraft object or rover MJCF

The description is either a Basilisk object with reaction-wheel and thruster tables or a rover model with steering and bogie joints, built from the design.

Execute

In the benchmark

05 · Control

Fault-adaptive spacecraft docking

Thruster wiring in the simulator is unknown and silently re-keyed, so the controller has to identify it in flight and dock.

06 · Environment

Orbital keep-out zones or lunar terrain

From a spec or terrain renders, the agent would build an orbital scene with keep-out zones or lunar terrain with waypoints, plus a success check.

07 · Behavior

Free-floating reach and rover waypoints

An actor would reach a target with a free-floating dual-arm robot or drive a rover to waypoints, running on the controller.

End to end · 01 → 07

08 · End to end

Design a lunar rover, then reach waypoints

One agent would build a rover, its stack and a terrain from a brief, servo and wheel CAD and terrain photos. The score would count the waypoints it reaches on hidden missions.

Underwater robot (UUV/AUV)

Workflow map

An underwater vehicle has no IK, so control starts with thrust allocation and identifying the hydrodynamics. The description carries a hydrodynamics block of added mass and damping, which would be checked by its step response in any engine.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Buoyancy, thrust and endurance ledger

A mission brief plus thruster and battery facts would settle the buoyancy, thrust and endurance margins, the thruster count and the BOM.

02 · Hardware design

Frame and thruster layout

Around enclosure and thruster stand-ins, the agent would lay out a frame whose thrusters push and turn it in every direction, with foam for stability.

Build

03 · Assembly

Thrusters mounted, housings closed

Mounting the thrusters on the frame and closing the enclosure housings would be an assembler robot's job, scripted by the agent's program.

04 · Robot description

Vehicle model with a hydrodynamics block

Buoyancy, added mass, damping and a thrust curve for each thruster go into an SDF or MJCF vehicle model built from the assembly.

Execute

05 · Control

Thrust allocation and autopilots

With the model and logs, the agent would identify the hydrodynamics, then write depth and heading autopilots and an allocator that respects thruster limits.

06 · Environment

Dock, seabed or pipeline world

Built from a task text or goal renders, the dock, seabed or pipeline world would have currents and turbidity, plus a success check and an oracle.

07 · Behavior

Docking under current

Under unknown currents, the actor would guide the vehicle into a funnel dock at low speed without touching the walls.

End to end · 01 → 07

08 · End to end

Design a UUV, then dock it in a current

From a mission brief and thruster CAD, one agent would build the UUV, its autopilot, a dock scene and the docking actor. The docking runs would face hidden currents.

Soft robot

Workflow map

A soft model has no joint axes to check, so it would be judged by mesh and FEM validity; it can be a tendon chain, Cosserat rod, SOFA FEM scene or voxel grid. Cast bodies skip assembly.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Actuation, material and pressure budget

The agent would fit an elastomer model to the material data, pick an actuation type for the fragile-object brief, and write a pressure budget and a BOM.

02 · Hardware design

Air chambers and their casting molds

Given the material model and a bending target, the agent would design PneuNet chambers and the molds that cast them, or a Fin-Ray finger in CAD.

Build

SkippedNot on this platform

03 · Assembly

No assembly for cast or voxel bodies

Cast and voxel bodies have no discrete parts, so the design passes straight to the robot description.

04 · Robot description

Finger FEM scene or flex body

A SOFA FEM scene or a MuJoCo flex body, made from the finger mesh and a material card, has to bend under pressure like the reference.

Execute

05 · Control

Pressure control of finger bending

A pressure controller on the model would track a commanded finger bend through the dynamics of its valves.

06 · Environment

Fragile-object grasp scene

Fragile objects would sit in a grasp-and-lift scene built from a task text and goal renders, with a success check and an oracle.

07 · Behavior

Fragile grasps without crushing

An actor on the pressure controller would pick up fragile objects of unseen size and mass without crushing them.

End to end · 01 → 07

08 · End to end

Make a soft gripper, then pick fragile objects

One agent would make a cast gripper and its stack from material data and a fragile-object brief, and the grader would have it pick up hidden objects without crushing them.

Surgical robot

Workflow map

The remote center of motion (RCM) runs through the chain: the instrument pivots about a fixed point at the incision, and both the model and the IK must hold that point.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Shaft, pivot and tip-force requirements

A procedure brief and a catalog would fix the shaft diameter, the RCM and port pose, the tip's workspace and force, and the BOM.

02 · Hardware design

Instrument wrist and RCM mechanism

In code-CAD, around actuator stand-ins, the agent would design a needle-driver wrist, an RCM mechanism or a tendon-driven continuum section.

Build

03 · Assembly

Wrist and RCM linkage pinned together

The agent's program would have an assembler robot join the printed links with pins, forming the wrist and RCM linkage in simulation.

04 · Robot description

Parallelogram model that keeps the RCM

For the pivot to hold, the parallelogram joints need mimic or equality constraints in a model built from meshes and DH facts.

Execute

05 · Control

RCM-constrained IK and servo interface

IK on the model would reach tooltip targets with the shaft on the pivot, behind a CRTK-style servo interface.

06 · Environment

Needle tray, peg board or phantom scene

Around the instrument, a tray, peg board or phantom scene would be built from a task text or a goal photo, with a success check and an oracle.

07 · Behavior

Needle pick and peg transfer

An actor using the IK would grasp a needle from the tray and lift it, or move pegs between posts, on unseen layouts.

End to end · 01 → 07

08 · End to end

Design an instrument, then pick up a needle

One agent would build the instrument, with a model and IK that keep the pivot, from a procedure brief and actuator CAD. Its needle pick would run on hidden poses.

Swarm

Workflow map

The robot is a team, so requirements set the team composition and the robot description is a fleet manifest. The environment has to spawn every unit, and control runs as a tracker on each robot.

This platform has no tasks yet; each stage shows what it would produce.

Design

01 · Requirements

Team composition and unit sizing

The agent would pick robot classes, counts and capabilities from a robot catalog to meet a mission brief, with a BOM for each unit class.

02 · Hardware design

Arena fixtures around the units

Each unit's body comes from its own platform, so the CAD here covers the arena's charging docks and marker mounts, designed around the units' stand-ins.

Build

03 · Assembly

Unit assembly, delegated to its platform

Units are assembled the way their own platforms assemble them, and the team adds no assembly step.

04 · Robot description

Unit model and fleet manifest

The unit design becomes one loadable unit model, and a fleet manifest lists uniquely named copies of it with non-overlapping spawn poses.

Execute

05 · Control

Per-robot trajectory tracker

Each robot runs a velocity and pose tracker, written for the unit model, along its own reference path.

06 · Environment

Warehouse with a multi-agent wrapper

A spec or top-down renders would describe a warehouse of shelves, stations and spawns, wrapped for multiple agents and paired with an oracle.

07 · Behavior

On-time deliveries by the fleet

Parcels have deadlines, and a planner and dispatcher, or per-robot actors, would run on the trackers to deliver them in time.

End to end · 01 → 07

08 · End to end

Design a fleet, then run a warehouse

One agent would turn part stand-ins, a mission brief and a warehouse render into a fleet and its stack; the grader would count on-time deliveries on hidden layouts.

03How a task runs

The ALE pipeline

Every task is one folder in Harbor’s task format and runs on ALE, the engine of Agents’ Last Exam. One command runs an agent in a sandbox, and the hidden grader comes in after the agent stops.

ale run tasks/space/fault_adaptive_spacecraft_docking --agent claude-code --model <model>

Task folder

task.toml(never reaches the agent)
The configuration: resources, time limits, network.
instruction.md
The prompt, and the only thing the agent is told:
“Build an actively identifying, fault-adaptive controller …”
environment/
The simulator and its tools, on an official ALE base image.
environment/setup/
Runs as root before the agent starts and stages its payload.
solution/(never reaches the agent)
The reference artifact. ale validate runs it in place of the agent.
tests/(never reaches the agent)
The grader and its anchor, kept out of the sandbox while the agent works.

Episode

ModelAny model
- Claude
- GPT
- Gemini
- Qwenand others
HarnessAny ALE harness
- Claude Code
- Codex CLI
- Grok Build
- OpenClaw
- Computer use
SandboxThe agent works at its own UID
- Docker container
- QEMU VM
Absent while the agent works
- tests/(never reaches the agent)
- solution/(never reaches the agent)
Artifact/home/user/submission/
- Controller
- Plan
- CAD · URDF · MJCF
- Code
- Checkpoint
- Report

Phase order

Provision
Setup
Agent
Evidence
Verify
Result

Verifier

Arrives after the agent has stopped
Runs its own simulator on hidden seeds
Computes the metric itself

Scoring

anchor.json of this task

metric: mean_docking_quality
value: 0.744 measured
full_at: 0.75

Outputs

reward.json
verification.json
result.json
lock.json

lock.json is the RunLock, the record of what produced the result.

Fig. 2One ale run of the benchmark’s spacecraft docking task.Never reaches the agent

What the agent leaves

The agent’s artifact is whatever it leaves in /home/user/submission/, in any form. Each task’s grader takes it in one of three patterns. The author picks the pattern inside verify.py; the manifest has no field for it.

Controllerin a loop with theGrader’s simulator
Closed loop
closed_loop
The agent leaves a controller process. The grader launches it at the agent’s UID and steps it over a stdio wire, one JSON object per line and one observation per step. The template ships this pattern.
```
{"type": "reset"}
  → {"ok": true}
{"type": "act", "t": 0, "obs": …}
  → {"action": …}
{"type": "close"}
  → {"ok": true}
```
Plan or designthenRe-simulated
Open loop
open_loop
Answers, plans or designs stay on disk when the agent is done. The grader re-simulates or measures them under its own dynamics and rejects an answer that does not start where it claims to.
Checkpoint or programthenRolled out
Artifact rollout
artifact_rollout
Here the artifact is a frozen checkpoint, a planner program or generated code. The grader rolls it out on hidden seeds inside deprivileged(…) at the agent’s UID, because loading or running it as root would give it control of the grader.

What keeps the scores honest

Stage isolation
tests/ and solution/ reach the sandbox only when their own stage runs. While the agent works, neither is anywhere in it.
An unprivileged solver
The agent’s solver runs at the agent’s UID. Only root can enter /opt/ale, where the grader runs.
Seeds in memory
The grader draws its hidden seeds from OS entropy inside its own process, and they never touch disk.
A computed metric
The metric comes from the grader’s simulator state. Nothing the agent prints is read as a score.
Zero and one
ale validate runs each task twice. An untouched sandbox has to score 0 and the oracle 1, with the reward rounded to two decimals.
A measured anchor
The anchor is the value the reference implementation measured in an ale validate run.

The repository is public, and each task folder in it is complete, grader and anchor included. Stage isolation keeps them out of the agent’s reach while it works. Read the security model

Contributing a task

Read the task template

Copy the template
templates/task/
The template is a complete task, with a null anchor until you measure one.
Lint it
ale lint
Checks the folder against the engine’s task format.
Validate it
ale validate
An untouched sandbox has to score 0 and the oracle 1.
Measure its difficulty
ale run
Run frontier models on it. The target is a score below about 0.5.
Open a pull request
tasks/<platform>/<name>/
It adds this one folder. A maintainer runs ale validate again before merging.