Apply structured verification and validation techniques to AI and machine learning systems, from dataset quality and model documentation through output validation, robustness testing and drift detection.
This four-hour module develops practical capability in AI/ML Verification & Validation, focusing on the evidence needed to assess whether data, models and model outputs behave as expected.
Participants validate datasets, review model cards, check for data leakage, build golden test sets, calculate performance metrics, test robustness under perturbed and out-of-distribution inputs, and examine regression and drift between model versions and data windows.
The module concludes with a Full ML Pipeline Validation Report combining data validation, model documentation review and dynamic model-performance evidence.
View the Full Software Testing & Verification Programme
What You Will Learn
Module 9 applies the programme’s verification mindset to AI and machine learning systems.
Participants learn how to assess not only model outputs, but also the quality and integrity of the data and documentation surrounding the model.
Topics
- Dataset schema validation with Great Expectations
- Data quality checks: missing values, outliers, class imbalance
- Model cards: intended use, training data, evaluation, limitations
- Data leakage detection and train/test split validation
- Reproducibility checks: random seed management and dependency pinning
- Model output validation against known-answer (golden) test sets
- Performance metrics: confusion matrix, precision/recall tradeoffs
- Robustness testing: input perturbation and out-of-distribution handling
- Regression testing between model versions
- Data drift and concept drift detection; inference latency testing
Lab 1: Data Validation & Model Card Review
Learning Objectives: Great Expectations, data quality checks, model cards, data leakage
Description: Validate a sample training dataset and review its accompanying model card for completeness.
Tasks
- Define a set of Great Expectations validation rules for a sample dataset.
- Run the validation and identify one missing-value and one outlier finding.
- Check the dataset for class imbalance.
- Verify no overlap exists between the train and test splits.
- Review a sample model card against its required sections.
- Identify one gap in the model card (e.g. a missing limitations section).
Extension Tasks
- Check the pipeline for correct random-seed management and dependency pinning.
- Identify one representativeness gap in the dataset relevant to bias/fairness.
Topics Covered
Data Validation, Data Quality Checks, Model Cards, Data Leakage
Open Source Recommendation
Great Expectations, an open-source Python data validation framework.
Lab 2: Model Output Validation, Robustness & Drift
Learning Objectives: Model output validation, robustness testing, drift detection, regression testing
Description: Validate a sample model’s predictions on a known-answer test set and test its robustness and drift behaviour.
Tasks
- Curate a small golden test set with known-correct outputs.
- Validate model predictions against the golden set and compute a confusion matrix.
- Compute precision and recall and discuss the tradeoff for the sample use case.
- Test model behaviour under a perturbed (noisy) input.
- Test model behaviour on an out-of-distribution input.
- Run a simple statistical data-drift check comparing two sample data windows.
Extension Tasks
- Compare two model versions on the regression test set and set an acceptable performance delta.
- Measure inference latency under a simulated load.
Topics Covered
Model Output Validation, Robustness Testing, Drift Detection, Regression Testing
Open Source Recommendation
scikit-learn’s evaluation utilities and the open-source Evidently library for drift detection.
Module 9 Coverage Matrix
| Topic | Lab 1 | Lab 2 |
|---|---|---|
| Data Validation | ✓ | |
| Model Card Review | ✓ | |
| Model Output Validation | ✓ | |
| Robustness Testing | ✓ | |
| Drift & Regression Testing | ✓ |
Module Project: Full ML Pipeline Validation Report
Full ML pipeline validation report: data validation and model card review for a sample dataset/model pair, plus a dynamic validation report covering performance metrics, robustness and drift detection.
The project brings together data-level, model-level and output-level verification evidence.
Participants progress through:
Dataset
↓
Schema & Quality Validation
↓
Model Card Review
↓
Golden Test Set
↓
Performance Metrics
↓
Robustness Testing
↓
Regression & Drift
↓
ML Validation Evidence
This helps distinguish AI/ML verification from simply reporting a model accuracy score.
From Traditional Software Testing to AI/ML Validation
Traditional software testing often starts from deterministic requirements and expected results.
AI/ML systems introduce additional verification concerns around:
- Training and evaluation data
- Dataset quality
- Model documentation
- Statistical performance
- Robustness to changing inputs
- Model-version regression
- Drift over time
Module 9 therefore expands the programme’s verification approach into systems where behaviour is influenced by both software and data.
Data Validation Before Model Validation
Model evaluation is only meaningful if the data being used is suitable for the intended validation activity.
The first lab therefore begins with:
- Schema validation
- Missing-value checks
- Outlier detection
- Class imbalance checks
- Train/test separation
Participants use Great Expectations to define validation rules and identify concrete data-quality findings.
The curriculum also introduces representativeness as an extension topic relevant to bias/fairness.
Keep that wording appropriately limited: this is not a full AI fairness or governance programme.
Model Cards and Verification Context
The curriculum includes model cards as part of the evidence surrounding a model.
Participants review:
- Intended use
- Training data
- Evaluation
- Limitations
and identify missing information in a sample model card.
This gives the technical testing activity context about what the model is intended to do and where limitations should be documented.
Golden Test Sets and Model Output Validation
Module 9 introduces a known-answer or golden test set as a basis for validating model predictions.
Participants compare predictions against known-correct outputs and compute:
- Confusion matrix
- Precision
- Recall
The purpose is not simply to generate metrics, but to interpret their tradeoffs within the sample use case.
Robustness and Out-of-Distribution Testing
A model may perform well on expected data yet behave differently when inputs change.
The curriculum therefore asks participants to test:
Perturbed input
A noisy or modified version of a normal input.
Out-of-distribution input
An input that differs from the distribution the model is expected to handle.
This extends model validation beyond nominal test-set performance.
Regression Testing Between Model Versions
Software regression concepts also apply when models change.
The curriculum includes comparing two model versions against the same regression test set and defining an acceptable performance delta.
This gives teams a structured way to evaluate whether a model update has materially changed previously validated behaviour.
Data Drift and Concept Drift
Module 9 introduces both:
- Data drift
- Concept drift
and includes a practical statistical data-drift comparison between two data windows.
How Module 9 Connects With Other Modules
Module 5: Requirements-Based Testing & Traceability
Module 5 establishes traceable expected behaviour and verification evidence. Module 9 applies similar evidence-based thinking to AI/ML systems, where data and model behaviour introduce additional validation dimensions.
Module 8: IoT & Networking Applications Testing
Module 8 focuses on connected application reliability. Module 9 shifts the verification boundary into AI/ML data and model behaviour.
Module 10: Automotive/ADAS Safety-Critical Testing
This is an important progression because Module 10 includes sensor fusion, object/lane detection and scenario-based ADAS testing.
Frequently Asked Questions
What does the AI/ML Verification & Validation module cover?
The module covers data validation, model cards, data leakage, reproducibility, golden test sets, performance metrics, robustness, regression and drift detection.
Which tool is used for dataset validation?
The curriculum uses Great Expectations for sample dataset validation.
What data-quality checks are included?
Participants check missing values, outliers and class imbalance and verify that training and test sets do not overlap.
Does the module cover model cards?
Yes. Participants review a model card for intended use, training data, evaluation and limitations, then identify a missing section or other gap.
What is a golden test set?
Within this module, participants curate a small known-answer test set with known-correct outputs and use it to validate model predictions.
Which model-performance metrics are covered?
The module includes confusion matrices and precision/recall tradeoffs.
Does the module cover robustness testing?
Yes. Participants test model behaviour under a perturbed input and an out-of-distribution input.
Is regression testing between model versions included?
Yes. An extension task compares two model versions against the regression test set and defines an acceptable performance delta.
Does the module cover data drift?
Yes. Participants perform a simple statistical data-drift check comparing two sample data windows.
Is inference latency covered?
Yes. Inference latency testing is included in the curriculum, with a simulated-load measurement as an extension task.
What is the Module 9 project?
The project is a Full ML Pipeline Validation Report combining dataset validation, model card review, performance metrics, robustness testing and drift detection.
