Coverage-Guided AI Testbench Generation: What CovR Means for Sign-Off

Coverage-guided AI testbench generation turns testbench writing from a one-shot coding task into a measurable optimisation loop. A model proposes stimulus, a simulator executes it, coverage tools show what remains unexplored, and the model tries again. CovR, presented at the 2026 ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD), shows how reinforcement learning and agentic self-reflection can make that loop substantially more effective [1].

The important result is not simply that an AI system produced more tests. The CovR authors fine-tuned small open models (Qwen3-4B and Qwen2.5-Coder-7B) using executable tool feedback. The tuned models achieved higher simulation validity and code coverage than their base versions on established register-transfer level (RTL) benchmarks: VerilogEval, RTLLM v2.0 and CVDP. When the resulting stimulus engine was plugged into two complete verification workflows, average coverage (cov@1) rose by 18.95 percentage points. Mutation detection improved by only 1.19 percentage points, however, and the stronger stimulus also exposed problems in generated reference models that earlier tests had missed.

That difference is the central engineering lesson. Coverage is valuable feedback for an AI agent, but it is still a proxy for verification progress. Sign-off depends on whether the environment exercises relevant behaviour, checks the right outcomes, exposes plausible defects and remains traceable to requirements. An optimiser can improve the metric it is given without establishing those wider conditions.

Key Takeaways

• Coverage is feedback, not a verdict. In CovR, average coverage rose 18.95 points while mutation detection rose 1.19.
• Tools, not an LLM judge, score the reward. Format, compilation and code coverage drive the reinforcement-learning signal, so a persuasive explanation cannot earn reward.
• Better stimulus exposes weaker checkers. Reference-model faults surfaced, so checker and oracle quality can become the next bottleneck.
• Sign-off still needs intent. Pair AI stimulus with functional coverage, mutation testing, qualified checkers and reviewed waivers.

How Coverage-Guided AI Testbench Generation Works in CovR

Coverage-directed test generation is not new. Earlier approaches, including genetic algorithms, steered stimulus towards uncovered points. What is new is that a language model can write the stimulus code itself and learn from simulator feedback.

Many LLM-based testbench systems generate code and then repair syntax or compilation errors. CovR goes further by treating test stimulus generation as a learning problem driven by simulator and coverage-tool output. Its workflow starts from a natural-language specification and the RTL design, generates a test plan and initial stimulus, and then runs two feedback loops.

1.  Execution loop: Compilation and runtime errors are passed to a reflector, curator and fixer so the testbench can be repaired until it executes or reaches an iteration limit.

2.  Coverage loop: Coverage reports identify unexercised branches, conditions, toggles and state transitions. The generator proposes new stimulus aimed at those gaps while retaining context from earlier attempts.

The researchers used this process, with GPT-4o-mini and DeepSeek-R1 as teacher models, to build 16,514 specification, RTL, reasoning and testbench tuples. They then applied supervised fine-tuning (SFT) followed by reinforcement learning with Group Relative Policy Optimization (GRPO). The reward is hierarchical: the output must have the expected format, then compile and execute, and only then earn a coverage reward. The published configuration weights format at 0.02, compilation at 0.13 and coverage at 0.85 [1].

This architecture matters because, in the reinforcement-learning stage, the reward comes from engineering tools rather than an LLM judging its own work. The authors used Synopsys VCS and URG for simulation and coverage evaluation, and the framework also supports open-source Icarus Verilog and Covered. Compilation, simulation and coverage reports provide repeatable feedback. The model can still misunderstand verification intent, but it cannot earn a high execution or coverage reward merely by writing a persuasive explanation.

Why the Reward Function Matters

Reinforcement learning changes model behaviour by rewarding observable outcomes. The choice of reward therefore defines what the system learns to pursue. CovR rewards format, successful execution and average code coverage. That is a defensible design for stimulus generation because malformed or non-executable tests receive little value, while tests that reach more implementation structure receive more.

The same design also creates a boundary. The reward does not directly measure whether the coverage model represents every requirement, whether the reference model is correct, whether a checker can detect the wrong result, or whether an unhit state is important and reachable. Those questions sit outside the optimiser unless engineers add suitable evidence and acceptance checks.

This distinction is familiar in conventional verification. Verification planning and coverage closure connect requirements, coverage, assertions, observability and closure criteria. AI does not remove that structure. It makes the structure more important because a capable optimiser will pursue whatever the workflow measures [2].

What the CovR Results Actually Show

CovR reports several results that should be read together. All figures below are the authors’ reported results and have not been independently reproduced. The table separates the evidence from the conclusions that would require additional work.

Table 1  Interpreting the principal CovR results in a verification context. Point changes are averages over four settings: CorrectBench and PRO-V-R1, each on VerilogEval + RTLLM v2.0 and on CVDP.

The contrast between the 18.95-point coverage gain and the 1.19-point mutation-score gain is especially useful. It does not mean the additional coverage was worthless. It means that exploration and defect sensitivity moved by different amounts, and that the mutation average hides mixed results across settings. That is exactly why verification teams should avoid compressing several forms of evidence into a single headline percentage.

Scope also matters. The benchmark designs are typically single-module RTL, and the mutation test used only ten mutants per design, so treat the numbers as evidence of direction, not as a prediction for your own IP.

Coverage Is Feedback Rather Than a Verdict

Code coverage reports whether simulation exercised implementation structures such as lines, branches, conditions, toggles and finite-state-machine transitions. It is valuable for locating areas that stimulus did not reach. It does not state whether the exercised behaviour was correct or whether the implementation structure corresponds to every important requirement.

Functional coverage addresses a different question: whether scenarios defined by the verification plan have been observed. Its value therefore depends on the quality and completeness of the coverage model. A poorly chosen set of bins can reach 100% while omitting a critical ordering rule, recovery path or security condition. CovR currently optimises code-coverage dimensions; the paper identifies expansion to functional coverage as future work [1].

Coverage can also rise without improving observability. A test may drive a rare branch but fail to compare the relevant output, update the scoreboard correctly or assert the required temporal relationship. The environment has reached the behaviour without establishing whether it was correct. For AI-generated stimulus, this creates a practical acceptance rule: every important coverage objective should have a corresponding checking strategy.

Figure 1  Coverage-aware AI should optimise stimulus inside a wider evidence loop. Coverage guides iteration; it does not determine release readiness.

Mutation Testing Adds a Second View

Mutation testing deliberately modifies RTL and asks whether the verification environment exposes the changed behaviour. A killed mutation shows that the selected stimulus and checking mechanisms distinguished that mutant from the expected design. It therefore tests the verification environment, not merely the DUT.

CovR used ten mutants per design, generated with MCY (Mutation Cover with Yosys), in its full-workflow evaluation. It reported the percentage of generated environments that detected all ten, a perfect mutation score. The average gain was smaller than the coverage gain. This is a useful warning against assuming that broader exploration automatically produces proportional growth in defect detection.

Mutation results still need context. Their value depends on the mutation operators, locations, design, oracle and checking architecture. A perfect kill rate for one selected mutation set is not proof that every realistic defect would be found. Equally, a surviving mutant is not always a verification gap, because some mutants are functionally equivalent to the original design. Benchmark work such as VeriBugBench derives its mutation operators from real RTL bug-fix histories, one way to make injected faults more representative [5]. Alpinum’s AlpinumDV mutation-testing analysis explores the same distinction using results from nine designs [3].

Checker and Oracle Quality Can Become the Next Bottleneck

CovR’s stronger stimulus reduced the proportion of generated verification environments that passed on bug-free RTL by an average of 4.46 percentage points. Because the RTL was known to be good, the authors interpret the drop as previously hidden faults in the generated functional reference models becoming visible. In other words, better stimulus challenged the oracle as well as the design.

This matters for end-to-end AI verification systems. If the same specification is used to generate both the design-facing stimulus and the expected-result model, shared misunderstandings can survive undetected (a common-mode failure). More coverage may eventually reach a case where the reference model is incomplete or wrong. The resulting mismatch is evidence that the verification infrastructure needs review; it is not automatically evidence of an RTL defect.

A trustworthy workflow must therefore qualify the checker. Teams should retain known-answer tests for the reference model, compare selected outcomes against an independent implementation, inject faults into the checker path where practical, and distinguish an RTL failure from an unverified reference. That distinction is already explicit in the published AlpinumDV workflow, which separates genuine RTL bugs from reference-unverified outcomes.

Five Evidence Layers for AI-Generated Stimulus

A practical evaluation should separate five layers. Each answers a different engineering question.

1.  Build and execution validity: Does the generated environment compile, run and terminate without infrastructure errors or timeouts?

2.  Exploration quality: Which code structures, functional scenarios, cross combinations and state transitions were exercised?

3.  Checking quality: Were outputs, protocols and temporal relationships compared with a sufficiently independent and reviewed oracle?

4.  Defect sensitivity: Can the environment expose seeded faults, historical bugs or controlled RTL mutations that represent relevant failure classes?

5.  Intent and sign-off context: Which requirements and risks does the evidence address, what remains unverified, and who approves assumptions, exclusions and waivers?

An AI testbench generator can contribute strongly to the first four layers. The fifth remains a programme-level engineering responsibility. This is consistent with Alpinum’s earlier analysis of why AI-assisted verification must remain connected to verification intent and evidence before sign-off [4].

A Practical Acceptance Workflow

Teams piloting coverage-guided AI can use the following acceptance workflow without changing ownership of the verification plan. For the wider governance questions, see Alpinum’s guide to piloting AI in design verification safely.

1.  Define the verification objective first: Select a bounded block, protocol or feature and identify the requirements, risks, coverage objectives and checking mechanisms before generation begins.

2.  Constrain the generation environment: Provide the specification, RTL interface, legal stimulus rules, reset behaviour and approved toolchain. Record model, prompt, configuration and source versions for reproducibility.

3.  Collect a portfolio of metrics: Measure execution validity, code coverage, functional coverage, assertion outcomes, scoreboard mismatches and regression stability. Do not allow one metric to conceal deterioration in another.

4.  Challenge the environment: Run controlled mutations, replay historical bugs and test known corner cases. Review both killed and surviving mutations to determine whether the limitation lies in stimulus, checking or the mutation model.

5.  Review unreachable and waived targets: Require an engineering explanation for exclusions. An AI agent may suggest that a target is unreachable, but the conclusion should be supported by design analysis or formal evidence.

6.  Approve evidence rather than output volume: Sign-off should consider traceability, residual risk, checker qualification and reviewed waivers. The number of generated tests or covered lines is not itself an approval criterion.

How Alpinum Applies This Principle

AlpinumDV follows a specification-driven workflow that generates a test plan, coverage model, assertions, reference model and a Universal Verification Methodology (UVM) environment [6] before compiling, simulating, debugging and targeting uncovered bins. Its published results report functional coverage and mutation kill rates separately and qualify their benchmark scope [3].

The wider AI in Design Verification service focuses on measurable integration with existing engineering processes. For teams strengthening the underlying methodology, Alpinum’s Design Verification Training with SystemVerilog and UVM covers checking, assertions, functional coverage, UVM, planning and sign-off. The Formal Verification Training extends that evidence model through property writing, proof analysis, formal coverage, completeness and over-constraint review.

Questions to Ask Before Adopting Coverage-Guided AI

• What exact metric is the agent optimising, and why is that metric connected to our verification objective?
• Can the system distinguish compilation success, coverage growth, checker failure and a genuine RTL defect?
• Are functional coverage and requirements traceability evaluated alongside code coverage?
• How are reference models and scoreboards qualified independently?
• Which mutation operators or historical bugs are used to test defect sensitivity?
• Can engineers reproduce the generated stimulus and inspect the evidence behind each iteration?
• Who approves unreachable targets, exclusions, waivers and the final sign-off decision?

Conclusion

CovR provides encouraging benchmark evidence that reinforcement learning, self-reflection and simulator-derived rewards can improve AI-generated RTL stimulus. Its results also show why coverage-aware generation should be evaluated as one part of a complete verification system. The 18.95-point coverage gain, 1.19-point mutation-score gain and newly exposed reference-model failures describe different aspects of performance. None should be used alone as a sign-off claim.

The most credible direction for AI in design verification is therefore controlled optimisation inside an evidence-led workflow. Let the model search for hard-to-reach behaviour. Let engineering tools score executable results. Then challenge the environment with qualified checkers, mutations, formal analysis and requirement-level review. Coverage can guide the loop; verification intent and engineering judgement must still decide when the evidence is sufficient.

Teams evaluating this approach can explore AlpinumDV or discuss a controlled AI in DV assessment with Alpinum.

References

[1] M. Abdelatty, M. Nouh and S. Reda, https://arxiv.org/abs/2609.19189 ,” 2026 ACM/IEEE International Symposium on Machine Learning for CAD (MLCAD), 2026. arXiv:2609.19189. DOI: https://doi.org/10.1145/3831599.3840347

[2] Alpinum Consulting, https://alpinumconsulting.com/blogs/verification/verification-planning-to-coverage-closure/

[3] Alpinum Consulting, https://alpinumconsulting.com/blogs/ai-ml-overview/alpinumdv-ai-uvm-mutation-testing-results/

[4] M. Bartley, Electronics World and Alpinum Consulting, 2026, https://alpinumconsulting.com/blogs/ai-ml-overview/electronics-world-ai-assisted-verification-intent-sign-off/  .

[5] X. Meng et al., arXiv preprint arXiv:2609.18022, 2026  https://arxiv.org/abs/2609.18022  .

[6] Accellera Systems Initiative, accellera.org  https://accellera.org/downloads/standards/uvm.

FAQs