Agents' Last Exam
for Robotics
The robotics domain of Agents' Last Exam. Autonomous agents are dropped into sandboxed robotics problems and scored on the artifact they leave behind — by a hidden grader, on hidden seeds.
Four commitments that make results meaningful
The Agentic Robotics Benchmark is built so that a number on the leaderboard actually means what it says.
One kind of task: agentic robotics
Each task drops an autonomous agent into a sandboxed robotics problem — develop a controller, plan a manipulation sequence, design a mechanism, write code-as-policy, tune a planner — and scores whatever artifact it leaves behind: a controller process, a plan, a design file, generated code. There is no answer key to retrieve.
Grader-computed metrics
The authoritative metric is computed by the task's verify stage on hidden seeds, in a stage the agent never sees. Nothing the agent prints is a scoring input.
Hidden by isolation, not by secrecy
Everything the grader needs — the anchor included — travels inside the task folder and is kept from the agent by stage isolation: verify/ is simply absent while the agent works. Anchors are published here.
Recomputed, then attested
Every upload is the engine's own run records. The server re-derives each score from them and rejects a bundle that disagrees with itself; validated is a consistency badge, and only an organisation re-run earns verified.
Organized by platform and research direction
Every task is classified by platform and research direction, and scores one hardware-invariant headline metric against the reference implementation's own measured anchor.
Humanoid
0Manipulation
0Multirobot
0Quadruped
0Soft
0Spacecraft
0Surgical
0UAV
1- drone_hoverControl
Underwater
0Vehicle
0Engine-native, end to end
The upstream ALE engine builds the task image, runs the agent with the network blocked, freezes /home/user/submission, then runs the task's verify stage on hidden seeds. ale-export packs the engine's own records into the bundle you upload. Running a task needs the ALE engine and base image, available today to partner teams granted access by AgentsLastExam.
Submit a run →A ratio to a real run
The grader's measurement is divided by the task's anchor — the value the reference implementation actually produced through the same grader — and capped. The release aggregate is a macro_mean over the whole task set; a task a run skipped counts as 0. Each task page shows its anchor, cap and formula.
Browse tasks →Honest about what was checked
The server re-derives every score from the engine records and rejects a bundle that disagrees with itself. That earns validated — a consistency badge. Verified is a maintainer attestation after an organisation re-run, and nothing in an upload can grant it. An upload something blocks from validation (no bound challenge, unknown task version, authoring harness) is kept as self-reported and stays off the board.
View leaderboard →Bring a robotics problem into the benchmark.
The benchmark grows through contributed tasks — each one a self-contained folder (instruction, image, reference oracle, hidden verifier) that must score exactly zero untouched and exactly one with its oracle before it counts. Build and gate the task locally with the ALE engine, then open the pull request yourself or upload the zipped folder here and let the bot open it; maintainers review it either way.