Claude Opus 5.568 runs
✓ 58solved correctly
✗ 10mistakes1 wrong answer · 9 ran out of turns
- Mistake rate
- 15% 10 of 68 runs
- Fixes its own errors
- 98% 122 of 125 tool errors
PLC 39/48CAD 7/8Docs 12/12
Physical AI lab
Can AI engineer
the physical world?
Long-horizon tasks (60–150 tool steps) on a real, delivered industrial oven: Siemens TIA Portal program, electrical drawings, acceptance protocols, the engineers' revision history.
Long-horizon hard bench
22 long tasks · 35–199 tool steps · pass = every item exact · rounds 1–2 · 1 run per task, 2 runs on borderline tasks · each task weighs the same · updated 2026-11-06Pass rate
Tasks finished with every item exact · Higher is better
Time per Task
Mean minutes per task · Lower is better
Cost per Task
Mean USD per task · Lower is better
Pass rate vs. Cost per Task
Tasks fully correct · mean USD per task
- Claude Opus 5.5
- GPT-6.1 Sol
- Gemini 4 Argon
- DeepSeek V4.1 Flash
- Most attractive quadrant
- Model Pareto frontier
Score by Domain
Mean score, 0–100 · Higher is better
- Claude Opus 5.5
- Gemini 4 Argon
- GPT-6.1 Sol
- DeepSeek V4.1 Flash
Results per Task
22 tasks · 4 models
| Task | Claude Opus 5.5 | Gemini 4 Argon | GPT-6.1 Sol | DeepSeek V4.1 Flash |
|---|---|---|---|---|
| Clone a fan chain end to end | Claude Opus 5.5 · 97 | Gemini 4 Argon · 46 | GPT-6.1 Sol · 46 | DeepSeek V4.1 Flash · 21 |
| Eight-part site change package | Claude Opus 5.5 · 100 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 36 | DeepSeek V4.1 Flash · 31 |
| Cross-reference all 48 outputs | Claude Opus 5.5 · 89 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 89 | DeepSeek V4.1 Flash · 0 |
| Plant job: symptoms, change, spec check | Claude Opus 5.5 · 100 | Gemini 4 Argon · 85 | GPT-6.1 Sol · 77 | DeepSeek V4.1 Flash · 5 |
| IO tag → networks, HMI, alarms | Claude Opus 5.5 · 100 | Gemini 4 Argon · 99 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 84 |
| IO tag → networks, HMI, alarms (code) | Claude Opus 5.5 · 100 | Gemini 4 Argon · 0 | GPT-6.1 Sol · 99 | DeepSeek V4.1 Flash · 0 |
| Screen object → tags, writers, IO | Claude Opus 5.5 · 97 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 90 |
| Screen object → tags, writers, IO (code) | Claude Opus 5.5 · 100 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 0 |
| IO, safety chain and change impact | Claude Opus 5.5 · 100 | Gemini 4 Argon · 50 | GPT-6.1 Sol · 99 | DeepSeek V4.1 Flash · 0 |
| FAT protocol coverage audit | Claude Opus 5.5 · 100 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 93 |
| Predict the FAT result | Claude Opus 5.5 · 98 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 98 | DeepSeek V4.1 Flash · 8 |
| Write the burner FAT section | Claude Opus 5.5 · 100 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 95 | DeepSeek V4.1 Flash · 8 |
| Process-limit audit vs manuals | Claude Opus 5.5 · 87 | Gemini 4 Argon · 95 | GPT-6.1 Sol · 90 | DeepSeek V4.1 Flash · 81 |
| Spare-parts integrity audit | Claude Opus 5.5 · 98 | Gemini 4 Argon · 1 | GPT-6.1 Sol · 98 | DeepSeek V4.1 Flash · 1 |
| Manufacturing release of the lift | Claude Opus 5.5 · 100 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 0 |
| Scissor mechanism review | Claude Opus 5.5 · 90 | Gemini 4 Argon · 95 | GPT-6.1 Sol · 88 | DeepSeek V4.1 Flash · 0 |
| Commission the conveyor | Claude Opus 5.5 · 100 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 48 |
| Commission the conveyor (rules unstated, code) | Claude Opus 5.5 · 100 | Gemini 4 Argon · 97 | GPT-6.1 Sol · 97 | DeepSeek V4.1 Flash · 42 |
| Commission the conveyor (rules unstated) | Claude Opus 5.5 · 98 | Gemini 4 Argon · 50 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 74 |
| Commission the oven | Claude Opus 5.5 · 99 | Gemini 4 Argon · 100 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 25 |
| Commission the oven (rules unstated, code) | Claude Opus 5.5 · 99 | Gemini 4 Argon · 89 | GPT-6.1 Sol · 100 | DeepSeek V4.1 Flash · 61 |
| Commission the oven (rules unstated) | Claude Opus 5.5 · 100 | Gemini 4 Argon · 49 | GPT-6.1 Sol · 97 | DeepSeek V4.1 Flash · 57 |
| Pass rate | 55% · score 98 | 50% · score 80 | 45% · score 91 | 0% · score 33 |
Full score (100) Partial (1–99) ZeroEach task scores 0–100. Overall score = mean over the tasks a model ran. A dash = not run yet. One run per task; borderline tasks have two.
IO, safety chain and change impactElectrical dossier109 turns$12 cap229 files · 6.6 MBOpus 5.5ArgonSolDeepSeek
Brief
The workspace holds the engineering data of one industrial oven as text (read INDEX.txt first): the electrical schematic (one file per PDF page), the PLC tag tables, program blocks and hardware list (TIA Portal XML/CSV), the HMI export, the HMI alarm texts and the signed software acceptance protocols (the IO list). The documents were made by different people at different times and are not consistent with each other. You prepare the commissioning dossier for the site acceptance test, in three parts.
3 parts, graded item by item · IO reconciliation (108 channels) + safety-chain trace + change impact of 6 re-addressed signals
schematic page readingXML tag tablesrule application over 108 channelssafety-chain tracing across sheetsLAD/FBD network numberingPLC->HMI tag linksterminal diagramscross-document reconciliation
Workspace
- acceptance_protocols/5 · 15 KBsoftware acceptance protocols (the signed IO list)
- hmi/17 · 3.0 MBWinCC HMI screens and HMI tag tables (Openness export)
screens/ 15tags/ 2 - plc/102 · 3.4 MBPLC tag tables (TIA Portal Openness export, SimaticML) · hardware configuration · network nodes · program blocks
blocks/ 81tag_tables/ 19 - schematic_EN/102 · 241 KBelectrical schematic (EPLAN)
- texts/2 · 9 KBHMI alarm texts EN / FR as delivered to the customer · HMI user texts EN / FR
- INDEX.txt2 KBworkspace guide: what each folder holds
Grader
eleclib.py420 linesgrade_elec.py394 lines- Deterministic, item by item against reference answers; no LLM judge.
Budget
- 109turns, stated in the prompt
- $12per-run cost cap (ours)
Outcomes
| Opus 5.5 | best 100 | $5.71/run | |
|---|---|---|---|
| Argon | best 99 | $1.56/run | |
| Sol | best 100 | $0.51/run | |
| DeepSeek | best 0 | $0.59/run |
Earlier tasks (mostly solved)138 runs · 20 tasks · medium effort
Solved on content: these fail only on a tight turn budget, so they are not in the headline.
FamilyOpus 5.5ArgonSolDeepSeek
Handover records2 tasks3/4 · 750/4 · 24not runnot run
pass, every item exact fail stopped by our cost cap2/7 = runs that pass · 82 = mean score
#ModelScoreFiles onlyWith codeCost / taskTime / task
1Claude Opus 5.5Anthropic · medium effort878193$1.4996s
2Gemini 4 ArgonGoogle · medium effort838087$1.45224s
Gemini 4 Argon70 runs
✓ 52solved correctly
✗ 18mistakes7 wrong answers · 6 ran out of turns · 5 stopped early
- Mistake rate
- 26% 18 of 70 runs
- Fixes its own errors
- 92% 68 of 74 tool errors
PLC 43/52CAD 2/6Docs 7/12
Accuracy, time and cost
Accuracy
Mean score out of 100. Higher is better.
Time per task
Average wall time in seconds. Lower is better.
Cost per task
Average estimated cost in USD. Lower is better.
Long ladder logic separates the models
With file tools only, Opus 5.5 often runs out of turns reading the oven logic. Argon reads in larger steps and finishes.
A code tool closes the gap
Given a sandboxed Python tool, both models solve the hard tasks, about 24× cheaper than reading by hand.
Where it fails
A few failures we keep coming back to: what the model said, and what is real.
Real Siemens TIA Portal V19The oven project the PLC tasks use, opened in TIA Portal.
Recreate the engineers' commits · Claude Opus 5.5
One realistic, terse change ticket in a chain of 6 real commits.
- Opus 5.5
- 0 of 71 I/O lines right in the two acceptance annexes
- Engineers' commit
- 71 I/O lines, re-addressed
Same chain, same files. Only the ticket wording differs.
Score on the same task
Ticket spelled out
0.97
Ticket as the engineer wrote it
0.11
Claude Opus 5.5
Asked to copy an alarm text exactly, it wrote its own. On all 4 drives.
- It wrote
- “Frequency converter error”
- The file says
- “Inverter error”
Everything else in the answer was right. One invented text makes it a fail.
All three models
A timing value hidden in the control program. Missed by every model: 0 of 9 runs.
- Every model
- No problem found
- Real
- The program stops the fans after 20 seconds. The manual says “after a few minutes”.
Across Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon. No run found it.
AuditedFailures are audited against the source files; 18 of 22 Opus failures were genuine model mistakes, none were knowledge gaps.
For research labs.
Evaluate on real engineering work.
EvalEvaluation accessRun your model on the long-horizon bench.
Full envThe full environmentWindows + Siemens TIA VM, or Docker.
2.1 TB128 real projectsThe raw engineering archive behind the bench.
243Commits of engineer history per projectEach commit → a recreate-the-diff task.
The machine, part by part
Hover a part: the real CAD shows through.
Opus 5.5 failed
Opus 5.5 failed
The project
The lift, in real CAD
A hydraulic scissor lift that raises 1,000 kg by 1.3 m, designed by a real engineering office. This is its own 3D model and its own cutting files. Turn it, take it apart, and see where a model got the design wrong.
- Load
- 1,000 kg
- Lift height
- 1.3 m
- Parts in the model
- 1,160
- Laser-cut parts
- 56
3D model
Drag to turn · Scroll to zoom · Hover a part for its name · Red pins mark a model's mistakes
If Argon's number were used:
- Deck measured at 5.0 mm
- Spec says 4.0 mm, so inspection fails
- A lift that meets spec is held back
With Opus 5.5's 4.0 mm: passes, ships (2/3 runs).
Argon flagged deck_thickness = false in 3/3 runs · runs/gemini-4-argon/lift_model_vs_spec
Who fixed it better
Two traps in the lift's files. Opus 5.5 saw through both.
Where the two traps sit · tap a pin
Tap pin 1 or 2 for the real value and what each model said.
What the models received
- lift_assembly.step.gzSTEP model · 65 MB · 197 products · 15,338 faces
- flat_patterns/*.dxf56 laser cutting files, with marking crosses
- technical_specification.mdSigned spec: 1,000 kg · 1,300 mm · guardrail 1,200 mm
- commercial_proposal.mdOlder offer: guardrail 1,100 mm · pit 500 mm
- parts_materials.csv · price_list.mdSteel grade and thickness per part · laser rates
566 files delivered: 172 part models, 44 assemblies, 136 drawings, 140 PDFs, 56 DXF, hydraulic calculations
1Deck plate. Real plate: 4.0 mm. A tiny raised patch fools a bounding box.
✓ Opus 5.5 2/3 runs✗ Argon 0/3 runs
runs/*/lift_model_vs_spec/python_run1-3 · graders/cad_reference.json
Opus 5.5
7,594 mm² ✓
Took away 365 mm² as holes. Real holes: 365 mm².
Argon
5,471 mm² ✗ −28 %
Took away 2,488 mm² as holes. Real holes: 365 mm².
2Lower flange. Which loops are holes? Fill = metal the model counted. Real: 7,594 mm².
✓ Opus 5.5 right✗ Argon wrong loops
runs/claude-opus-5-5/lift_sheet_order/python_run1 · runs/gemini-4-argon/lift_sheet_order/python_run1 · TTN101.81.03.002.dxf · hover a flange for the numbers
Coming soonPhysical verification.
Physical verification.
Scored on real machines.
Today every answer is checked against the original engineering files. Next, the same tasks run on real hardware: a controller on a test rig, parts cut from the model's own drawings.
- PLC programs on a real controller hardware in the loop
- CAD parts cut and assembled fit and tolerance
- Robot tasks on a bench pick, place, assemble