Can AI-Generated UVM Testbenches Detect Controlled RTL Changes? New AlpinumDV Mutation Results

Generating a UVM testbench with AI is one challenge. Evaluating whether the resulting verification environment can detect deliberately introduced RTL changes is another. That distinction matters as AI becomes more involved in semiconductor design and verification workflows. Producing verification code quickly can reduce setup effort, but verification engineers still need evidence that an environment is exercising defined behaviour, checking expected outcomes and identifying mismatches when behaviour changes.

Our latest AlpinumDV 2.0.0 evaluation was designed to examine this more closely.

Across a specific nine-design evaluation completed in August 2026:

• Six of nine designs reached a 100% mutation kill rate for the mutation sets evaluated
• RISC-V CPU functional coverage increased from 58% to 85%
• SPI achieved a 100% mutation kill rate after an API-limit blocker affecting the previous run was resolved
• APB slave reached a 100% mutation kill rate, compared with 80% in the previous evaluation
• Scoreboards remained clean across all nine designs in the functional run

The objective was not simply to ask whether AlpinumDV could generate and execute verification environments.

We wanted to examine another aspect of verification effectiveness:

Could the generated environment expose the controlled RTL mutations introduced into the evaluated designs?

Benchmark scope: These results apply to this specific August 2026 nine-design AlpinumDV evaluation. They should not be interpreted as universal performance figures for every RTL design, mutation set or verification environment.

From AI-Generated Testbenches to Measurable Verification

AlpinumDV 2.0.0 follows a structured verification workflow. Starting from a specification and RTL where available, the current workflow supports verification planning, functional coverage and assertion generation, reference modelling, UVM environment generation, simulation, testbench self-debug and coverage improvement.

These capabilities sit within Alpinum’s wider work in AI in Design Verification, where the objective is not simply to generate engineering artefacts but to make AI-assisted outputs measurable and reviewable within established verification workflows.

Automation can reduce repetitive setup work. But automation alone does not establish confidence in verification. A verification environment that compiles successfully is not necessarily checking the right behaviour. A testbench that generates many transactions is not necessarily exercising every relevant scenario. And high functional coverage should not automatically be interpreted as proof that every relevant requirement, defect or behaviour has been tested.

This is one reason the latest AlpinumDV evaluation also included mutation testing.

For engineers developing the underlying methodology, Alpinum’s Design Verification for SV/UVM Training covers simulation-based verification, checking, assertions, functional coverage, UVM and practical verification strategy.

What Is Mutation Testing in RTL Verification?

Mutation testing introduces controlled modifications into a design and evaluates whether the verification environment distinguishes the modified behaviour from the expected behaviour.

Conceptually, the question is straightforward:

If we deliberately alter something in the RTL, does the verification environment expose the change?

When a particular mutation is exposed by the verification environment, that mutation is considered killed. The mutation kill rate therefore provides an additional way to assess how effectively the evaluated verification environment detects the specific RTL mutations used in the experiment.

It is important to be precise about what that means. A killed mutation is not automatically equivalent to finding a real production bug. Mutation testing deliberately introduces design changes so that the verification environment itself can be evaluated.

It is also not a replacement for:

• Functional coverage
• Assertions
• Code coverage
• Formal verification
• Requirements traceability
• Engineering review
• Sign-off criteria

Instead, it provides another form of evidence. Functional coverage asks whether defined verification scenarios and coverage points have been exercised. Mutation testing asks whether the environment can expose the deliberately modified behaviours represented by the selected set of mutations. Used together, the two measures provide different perspectives on the evaluated verification environment.

Nine Designs in the August AlpinumDV Evaluation

The evaluation included:

adder, counter, single-port RAM, dual-port RAM, FIFO, APB slave, SPI, ALU and a RISC-V CPU.

The set therefore includes combinational and sequential logic, memories, FIFO behaviour, bus/protocol logic, serial communication and processor behaviour. Across the functional run, the scoreboards remained clean on all nine evaluated designs. 

Several designs maintained 100% functional coverage, while APB slave and ALU reached 100% in the final run. The largest increase in functional coverage occurred on the RISC-V CPU.

RISC-V CPU: Functional Coverage 58% → 85%

RISC-V CPU functional coverage increased by 27 percentage points, from 58% to 85%, in the final run. This result does not establish verification completeness. This shows that the final environment exercised a larger proportion of the defined functional coverage model used for this particular RISC-V evaluation. Processor verification also introduces instruction execution, architectural state, and processor-specific behaviours that differ from those in smaller block-level examples.

For engineers working specifically on processor and SoC verification, Alpinum’s RISC-V Verification Training covers RISC-V CPU verification and CPU integration into SoC-level verification workflows.

Six Designs Reached 100% Mutation Kill Rate

Six of the nine evaluated designs achieved a 100% mutation kill rate within the mutation sets used for this benchmark:

• adder
• counter
• single-port RAM
• ALU
• APB slave
• SPI

The remaining final mutation kill rates were:

• dual-port RAM: 90%
• FIFO: 90%
• RISC-V CPU: 80%

Publishing individual results is more informative than presenting a single aggregate percentage because it shows how outcomes varied across designs.

SPI: Previously Constrained Run → Final 100%

The previous SPI mutation run was constrained to 60% due to an API-limit blocker. After that blocker was resolved, the final run completed the intended evaluation and achieved a 100% mutation kill rate for the mutation set used in that SPI run.

This should not be interpreted as a straightforward 40-percentage-point improvement in the underlying verification algorithm because the earlier result came from a constrained run rather than an equivalent completed evaluation.

The relevant final result is:

All mutations included in the completed SPI evaluation were killed.

APB Slave: 80% → 100%

APB slave reached a 100% mutation kill rate in the final evaluation, compared with 80% in the previous run. Its functional coverage also increased from 96% to 100%. These two percentages should not be treated as measurements of the same thing. Functional coverage describes how much of the defined coverage model was exercised. Mutation kill rate describes how the verification environment responded to the deliberately introduced mutations during evaluation.

Together, they provide complementary evidence about the evaluated environment.

RISC-V CPU: Previously Constrained Run → Final 80%

The previous RISC-V CPU mutation run was also affected by the API-limit blocker and had stopped at 60%. Once the blocker was removed, the completed final run achieved an 80% mutation kill rate for the evaluated mutation set. Combined with the increase in functional coverage from 58% to 85%, this provides two distinct views of the RISC-V environment.

One measures progress against the defined functional-coverage model. The other measures detection of the controlled RTL mutations used in this benchmark. Neither figure should be treated as a standalone statement of verification completeness.

Why Publish the Individual Results?

AI-assisted engineering becomes easier to evaluate when the measurements, and their limitations, are visible. Different designs have different architectures, behaviours, verification objectives, coverage models and mutation sets. In this evaluation, final mutation kill rates ranged from 80% for the RISC-V CPU to 100% for six of the nine designs. Functional coverage similarly ranged from 85% on the RISC-V CPU to 100% on several other designs.

Publishing that variation is important. It shows where the evaluated verification environments performed strongly while also making clear that a single number should not be assumed to represent every design AlpinumDV may encounter.

What These Results Do — and Do Not — Prove

These results provide evidence from a specific AlpinumDV evaluation. They do not demonstrate that AI has solved the problem of design verification. They do not demonstrate that a particular mutation kill rate equals tape-out readiness. They do not demonstrate that every real-world defect would be detected. And they do not remove the need for engineers to define, review and own verification intent.

Mutation results depend on factors including:

• The mutation set
• The design under test
• The verification environment
• Checking mechanisms
• Stimulus
• Coverage models
• Assumptions and exclusions
• Acceptance criteria

Functional coverage similarly depends on the completeness and quality of the defined coverage model. Neither metric should therefore become an isolated sign-off criterion. Verification engineers still need to own requirements, verification intent, coverage strategy, assumptions, exclusions, acceptance criteria and final engineering judgement.

This is consistent with Alpinum’s wider approach to piloting AI safely in design verification: use AI where it can support engineering productivity while keeping outputs inspectable, measurable and subject to technical review.

The goal is not to replace verification methodology with an AI-generated number. It is to investigate where AI can automate parts of the workflow while keeping the resulting verification evidence reviewable by engineers.

Why This Direction Matters for AI in Design Verification

Much of the discussion around AI in verification begins with generation:

• Can AI write SystemVerilog?
• Can it generate UVM components?
• Can it propose assertions?
• Can it generate stimulus?

Those are useful questions. But they are not sufficient on their own.

Verification teams also need to ask:

• What verification objective is the generated artefact supporting?
• What behaviour did the environment exercise?
• What was actually checked?
• Which controlled mutations could the environment expose?
• What remained uncovered or undetected?
• Can an engineer review the evidence behind the result?

These are the kinds of questions AlpinumDV is being developed to address.

The direction is therefore not simply:

Generate more verification code.

It is:

Generate → Execute → Measure → Review

That distinction matters if AI-assisted verification is to become part of engineering workflows in which evidence, traceability and technical judgement remain important.

What Comes Next

This evaluation is another development data point for AlpinumDV rather than an endpoint. The ongoing focus is on making AI-assisted verification workflows increasingly structured, measurable and reviewable.

That includes continued work around:

• Functional coverage
• Checking quality
• Mutation-based evaluation
• Traceability
• Failure analysis
• Verification evidence

For teams considering AI inside existing semiconductor verification flows, generation speed can be valuable. But generation needs to be accompanied by evidence that engineers can inspect, interpret and challenge.

Explore AlpinumDV

AlpinumDV 2.0.0 supports a specification-driven workflow covering verification planning, reference modelling, UVM generation, simulation, testbench self-debug and coverage improvement.

Teams can also explore AlpinumDV examples through Alpinum’s Online Submission Portal, which includes demonstrations covering test-plan generation, UVM support, coverage analysis, reference modelling, APB, SPI, and RISC-V verification workflows.

Explore AlpinumDV 2.0.0

Run AlpinumDV Online

Explore AI in Design Verification Services

FAQs