Applied AI Read: What Build-A-Chip-Bench Tells Teams About Putting Design Agents to Work End-to-End

Teams rarely adopt an AI agent across a whole pipeline on day one. They adopt it one artifact at a time: a generated test case, a debug suggestion, a block of RTL. The operational problem shows up later, when the hand-off between stages has to survive without a human re-checking every step. That gap is what ChipAgents set out to measure with Build-A-Chip-Bench, developed with NVIDIA to evaluate AI agents across the end-to-end digital IC design flow, from specification through physical implementation and a verified Graphic Design System (GDS) result.

Why this particular flow makes a demanding test for applied work: custom silicon demand keeps climbing because physical AI, data centers, automotive systems, and edge devices require architectures tuned to particular workloads, memory bandwidth, latency, and energy efficiency. Producing such designs means coordinating heterogeneous compute blocks, complex interconnects, and multiple domains while meeting increasingly demanding power, performance, and area (PPA) targets. AI can expand engineering capacity and accelerate chip design cycles — design and verification engineers already apply it to generate test cases, assist debugging, and implement portions of RTL — so the next practical question is whether those capabilities extend across an entire flow, where decisions made during RTL implementation must hold up through verification, synthesis, and physical design.

Step 1 — See what the new benchmark adds to tools you already use

Benchmarks such as CVDP and ChipBench advanced AI for chip design by offering common tasks for evaluating and comparing systems. As frontier models improved, that style of measurement stopped separating two very different capabilities: doing an individual task well, and executing a complete design flow. Build-A-Chip Bench raises the bar on exactly those two dimensions — task difficulty and end-to-end scope — by asking agents to carry a design from workload specification through to final GDS, linking stages that narrower benchmarks evaluate in isolation.

For anyone deciding where an agent may operate, the consequence is blunt: correct RTL generation alone is not enough. An agent must preserve functional correctness, satisfy PPA constraints, and resolve problems that emerge during synthesis and physical implementation.

Figure 1. Reported pass@3 results for CVDP and ChipBench. CVDP scores plateau near 65–70%, with many remaining tasks appearing to have quality issues — ambiguous specifications or malformed testbenches — which limits measurable headroom. The best score to date is shown by orange lines; other model launches are shown by gray markers.

Step 2 — Follow the task path an agent is handed

At moderately large scale (designs of 100K-5M standard cells), the tasks exercise a representative specification-to-GDS flow and the dependencies across architecture, RTL, verification, and physical implementation. In industry those activities typically involve teams of architects, RTL designers, design verification (DV) engineers, physical design (PD) engineers, EDA engineers, and more; the benchmark requires a single agent to manage the technical dependencies across those disciplines.

1. Inputs provided. Each task gives a workload specification, a fixed interface, and an open-source EDA environment.
2. Scope the agent must own. Architecture, microarchitecture, RTL, verification, and physical implementation.
3. Constraint setting. Achievable constraints are established by reference designs, so the target is realistic rather than arbitrary.
4. Grading you cannot tune. Hidden verification IP evaluates functional behavior against held-out tests, while a separate grader evaluates the final submission.
5. Example task. An INT4 inference accelerator for Qwen3-0.6B must be built using Verilator, OpenSTA, OpenROAD, and Nangate45 PDK, meeting a 400 MHz timing target, fitting within 6 mm², and delivering at least 15 decode tokens per second at less than 10 W, with memory bandwidth pinned at 12.8 GB/s.
6. Decisions left to the agent. It chooses the datapath architecture, buffering, and scheduling; beyond RTL, it writes testbenches, creates EDA flow scripts, and performs manual floorplanning and placement as needed.

Figure 2. A representative specification-to-GDS flow with feedback paths. The benchmark checks do not establish production signoff.

Step 3 — Read the scoring model before you trust a demo

Fixed open-weight harnesses were used to evaluate all models. To pass, a design must clear two gates: functional verification against the hidden VIP, and a physical gate requiring a routed GDS layout with no hold-time or design-rule violations. Designs that pass both are then scored on PPA relative to task constraints, failures receive zero, and the overall score is the average across all tasks.

Model	Functional pass rate	GDS pass rate	Overall score	Application takeaway
Claude Fable 5.1	65%	75%	16.6%	Highest overall score in these runs; balance across gates matters more than any single rate.
GPT-6 Astra	85%	85%	8.7%	Led both pass rates, but a lower PPA score pulled the overall result down.

These preliminary results do not establish production signoff or silicon validation, and they should be read as a description of behavior under a specific toolchain and resource budget rather than as a product comparison.

Step 4 — Study where agents still stall, because that is where humans stay

Useful engineering judgment appears alongside inconsistent completion. One run showed an agent applying a post-route engineering change order (ECO), resizing one instance to eliminate a remaining slew violation, then passing final-netlist equivalence checks. Yet attempts to improve PPA can introduce functional regressions and consume the remaining time budget in debugging.

Architectural corrections can also be delayed while physical-design parameters are adjusted. One run showed an agent documenting a storage change that would reduce SRAM macros from 51 to 27, deferring it, and spending roughly six hours trying routing configurations before implementing it. Effective closure requires tracing physical failures back to the design decisions that produced them.

Iteration is further limited by tool runtime. A single place-and-route run can exceed four hours, so agents must prioritize changes and reserve sufficient time for final verification and implementation. The score plateaus observed in these sample runs reflect the same failure modes: functional regressions during optimization, delayed architectural corrections, and slow tool loops that stall further improvement (Figure 3).

Figure 3. Best agent-measured scores so far on two sample tasks during the first 36 hours of 72-hour runs, before hidden-VIP grading. Saved designs are marked by circles; failed attempts are marked by crosses. Best-so-far curves do not show every intermediate regression. Models absent from a panel produced no passing design on that task.

Common misconceptions to correct before rollout

• “A correct RTL dump means the task is finished.” The benchmark requires functional correctness, PPA satisfaction, and resolution of problems that emerge during synthesis and physical implementation.
• “The model with the highest pass rate wins.” GPT-6 Astra led both pass rates at 85% but finished at 8.7% overall because of a lower PPA score.
• “Optimizing always moves the design forward.” Attempts to improve PPA can introduce functional regressions and eat the remaining time budget.
• “A physical failure is a physical problem.” Closure depends on tracing physical failures back to the design decisions that produced them — as in the deferred change reducing SRAM macros from 51 to 27.
• “Agents can iterate as often as needed.” A single place-and-route run can exceed four hours, which caps how much iteration any workflow can afford.
• “Public benchmark headroom equals real remaining headroom.” CVDP plateauing near 65–70% partly reflects tasks with ambiguous specifications or malformed testbenches.
• “A passing benchmark run is signoff.” The benchmark spans the design flow but captures only part of the complexity of a production silicon program.

Action recommendations for application owners

1. Gate every stage. Require that work completed at any stage remains valid in subsequent stages before allowing the agent to move on.
2. Measure three behaviors, not one. Evaluation should measure whether agents preserve verified behavior during optimization, identify architectural causes of physical failures, and allocate runtime effectively.
3. Put a deadline on structural decisions. When an agent documents an architectural change, require it before long parameter sweeps — the roughly six hours spent on routing configurations in the observed run is time taken from closure.
4. Build the schedule around tool loops. Since a single place-and-route run can exceed four hours, reserve sufficient time for final verification and implementation instead of assuming unlimited retries.
5. Match the benchmark to the adoption question. Use narrower task sets such as CVDP and ChipBench when comparing individual steps; use Build-A-Chip Bench when the question is whether a complete specification-to-GDS flow can be delegated.
6. Keep engineers on decisions and validation. Under the tested toolchain and resource budgets, frontier agents demonstrated the ability to execute the specification-to-GDS flow but still struggled to preserve correctness and converge on strong PPA — the demonstrated capabilities point to substantial potential for AI-assisted silicon design, with engineers guiding design decisions and validating the resulting implementations.

The practical lesson for teams applying AI beyond chip design is the same: a stage-level demo tells you what an agent can produce, while an end-to-end measurement tells you what it can be trusted to carry. Score the hand-offs — preserved behavior, cause-traced failures, and runtime allocated — before you widen an agent's authority, and keep a human checkpoint wherever a stage can silently invalidate the work that follows.