GPT-6 Astra, the strongest of ten AI models put through a new web-development test called Co-Design Bench, delivered a fully correct result on just 26.4% of its tasks. On average it met about 70% of each task’s written requirements, which sounds respectable until the unmet 30% lands on a human developer’s desk. Three of every four jobs came back needing cleanup.
Catch up on AI in 3 minutes a day: 700,000+ readers get The Neuron, a daily email with the latest AI tools, tutorials and news.
We may earn a commission if you sign up.
Co-Design Bench was built by a team led by Xilin Wei, with colleagues from Alibaba and several Asian universities, to measure something older benchmarks skip: whether an agent can turn a real person’s messy change request into a working revision of an existing web app. Its scores look nothing like the near-perfect marks those earlier tests produce.
Co-Design Bench: 273 tasks mined from real coding sessions
The paper on arXiv says the 273 tasks were curated from 37,463 deduplicated queries drawn from 26,314 real user coding sessions. Requests favor the cases where a user had to rephrase an instruction or roll back a change, because those are the moments when an assistant has plainly missed. Some requests arrive with screenshots, audio, video or documents attached, so the agent must read more than text.
The tasks split into two tracks. According to the project page, 100 belong to Multimodal Editing and 173 to Repository Editing, with jobs ranging from a small visual tweak to a full overhaul of a project. Each task carries hidden acceptance criteria written by experts, 2,331 of them in total, and an AI judge opens the finished app in a browser and checks the evidence in the source code.
A second AI judge then challenges every passing verdict, hunting for mock-ups or hard-coded results that fake a feature. The design has a limit the authors acknowledge: each task has one fixed request, so follow-up questions and mid-task corrections are never tested, and both judges can still be wrong. The paper reports the ten models were run under one shared agentic harness, so differences between them are not simply differences in tooling, though the Qwen3.8-27B result suggests the harness can matter a great deal.
GPT-6 Astra’s 70% average against its 26.4% full-pass rate
The abstract states that the best endpoint “fully satisfies only 26.4% of tasks,” while passing 69.9% of the acceptance criteria per task on average. That gap is the entire story: partial credit is easy to earn, and a finished job is hard. The Decoder’s report on the results names GPT-6 Astra as the leader, with Claude Opus 5 second and Qwen3.8-Max third, and Grok 4.6 in last place.
Breaking GPT-6 Astra down by track shows how lopsided the picture is. It fully solves 41% of the multimodal tasks but only just under 18% of the repository tasks, where the agent must work through an entire codebase. Its project-page strict-completion scores were 75.4% on multimodal edits and 66.7% on repository edits, so even the track it handles best leaves a quarter of the criteria unmet.
Other results are less intuitive. GLM-5.2 consumed roughly six times as many tokens as GPT-6 Astra on smaller changes yet scored far worse, and the open-weight Qwen3.8-27B placed second on the larger overhauls, scoring 75% or 56% depending on which harness, the program that lets an agent read, write and run code, was wrapped around it. Every one of the ten models also fell back when a task required three or four kinds of change at once, such as layout, design, interaction and content.
Older web-coding scores above 99% and the merge-rate gap
Earlier tests of web editing have been close to maxed out. Models score above 99% on benchmarks such as the editing portion of WebCompass, the multimodal suite described in its arXiv paper, and Co-Design Bench shows a gap of more than 35 percentage points against those results. A benchmark that nearly everyone aces tells buyers little about how an agent handles an ambiguous instruction from a customer.
The finding echoes a separate warning from METR. Its March 2026 note had four active maintainers from scikit-learn, Sphinx and pytest review 296 AI-written patches that passed SWE-bench Verified’s automated grader, and found that roughly half would not have been merged. After normalizing against merged human work, maintainer acceptance ran about 24 percentage points below the automated scores. METR also cautioned that each agent got a single attempt with no feedback.
The cases chosen are the ones where a person was dissatisfied enough to retry, so the benchmark samples the hard end of ordinary use instead of the clean, well-specified prompts that fill most leaderboards. That selection is deliberate, and it is the main reason the full-pass rates are low. Whether the same ranking holds on a gentler mix of requests is something the authors do not claim.
The test of an agent’s worth in practice is the share of requests that need no human cleanup, and Co-Design Bench puts that share at 26.4% for the best of ten. What it cannot yet say is how much of the remaining distance a follow-up exchange would close, since the benchmark never lets a user reply to the agent’s first attempt.
This article was produced with the assistance of AI and reviewed by Morning Overview editors prior to publication.
Catch up on AI in 3 minutes a day: Join The Neuron →
We may earn a commission if you sign up.
More from Morning Overview
- The NSA says three phone features should be off whenever you aren’t using them
- Hurricane Hunter radar shows four warning signs that a tropical cyclone is about to strengthen, a University of Miami study found
- Tropical Storm Rachel is dumping up to 12 inches on four Mexican states on its way to major hurricane strength
- The FTC says Lens.com doubled the price shoppers saw in Google ads
