Exam Preparation: ISTQB CT-AI (v2.0)

Define AI as a system's ability to acquire, process, and apply knowledge and skills, then illustrate how modern AI analyzes data, detects trends, and makes data-driven decisions. Highlight the AI effect, showing that what counts as AI changes with perception over time.

Contrast conventional, rule-based systems with AI-based, data-driven models; learn how explicit logic yields predictability and transparency, while AI models introduce black-box decisions and testing challenges.

AI technologies form a toolbox of methods, combining fuzzy logic, search and reasoning techniques with machine learning to solve diverse real-world problems.

Explore how AI development frameworks support data preparation, selecting algorithms, and compiling models. Compare frameworks like MXNet, CNTK, IBM Watson Studio, Keras, PyTorch, scikit-learn, and TensorFlow and their processor needs.

Train in the cloud with high compute, then deploy to edge devices for inference, using GPUs for parallel ML workloads and AI chips for efficient edge processing.

Pre-trained models reduce cost and time by reusing existing AI, with transfer learning and fine-tuning for new tasks, while posing bias and transparency risks.

Discover how global and regional AI standards and regulations drive reliable, safe, and ethical systems, and why testers verify data accuracy, fairness, and compliance.

Explore the difference between flexibility and adaptability in AI systems, where flexibility absorbs unforeseen changes through generalization, and adaptability enables retraining, fine-tuning, or updating components, guided by requirements and testing.

Explore how evolution enables AI systems to continuously improve performance and efficiency. Learn from decisions and environmental changes while keeping evolution constrained to align with human expectations.

Explore ethics in AI, focusing on fair, responsible, and human-centered systems that benefit people and the planet, guided by OECD principles of transparency, safety, and accountability.

Explore side effects and reward hacking, showing how narrowly defined goals cause unintended outcomes and how testing reveals both results and the methods used.

Ensure safety in ai by preventing harm to people, property, or the environment in life-critical domains such as medicine, manufacturing, defence, and transportation, while addressing complexity and non-determinism with testing.

Explore the entire machine learning workflow as an iterative loop from defining objectives and success criteria to data preparation, model training, evaluation, deployment, and continuous monitoring.

Apply a decision framework to select supervised or unsupervised learning based on data quality and labels, then use classification, regression, clustering, or association, and consider reinforcement learning for problems.

Evaluate options based on problem functionality, data type, data amount, features, and trade-offs when selecting a machine learning algorithm. Rely on experience and experimentation because no single best algorithm exists.

Explore overfitting and underfitting, two key problems that affect how models generalize from training data to unseen data, and learn to balance complexity and data quality.

Explore how data preparation drives the ML workflow, guiding data acquisition, labeling, cleaning, transformation, augmentation, and feature engineering for training and predictions.

Identify and manage the major challenges in data preparation, from domain and technical knowledge to data quality across multiple sources, automation, scalability, and cost, to protect model accuracy.

Split data into training, validation, and test sets to reflect production and evaluate on unseen data. Use 60/20/20 or 80/10/10 splits and apply cross-validation to handle data and prevent leakage.

Data quality drives AI outcomes; even strong models falter with poor data. Testers must identify common dataset issues like wrong data, incomplete data, invisible labels, insufficient data, and unbalanced data.

Boost model quality by ensuring data quality drives accuracy, addressing bias and privacy risks. Testers validate data behind AI systems to prevent inaccurate predictions and data leakage.

Explore data labeling for supervised learning, from simple category labels to image annotation with bounding boxes, and diverse labeling approaches, while stressing data quality, validation, and mislabeling risks.

Explore how the confusion matrix reveals classifier mistakes, including true positives, true negatives, false positives, and false negatives, and how accuracy, precision, recall, and F1 assess performance.

Explore how performance metrics vary by problem type, from classification with ROC curves and AUC to regression with mean squared error and R-squared, and clustering with silhouette coefficient.

Understand what ml functional performance metrics measure—prediction accuracy, precision, recall, mse, auc— and recognize their limitations, including labeling quality, data bias, scope to the model, and tool dependency.

Compare machine learning technologies using benchmark suites with standardized tasks and evaluation criteria to measure training time and inference time across ML Commons, DawnBench, and MLMark benchmarks.

Imitate the brain with artificial neurons in input, hidden, and output layers that learn patterns from data through weights, biases, activation functions, trained by forward passes and error-driven weight updates.

Explore why traditional code coverage fails for neural networks and how neuron activation, threshold, sign-change, value-change, sign-sign, and mCDC-like metrics offer evolving coverage insights for testing.

Define clear requirements and testable specifications for ai-based systems, then compare expected versus actual behavior while addressing the test oracle problem with data-driven, probabilistic outputs and acceptance criteria.

Explore test levels for ai-based systems, focusing on input data testing and ml model testing, plus integration, system, and acceptance testing across data pipelines and performance criteria.

Prioritize test data as central to ai testing, using large, high-quality datasets that reflect real-world conditions, evolving data, and independence in testing to avoid masking defects.

Explore automation bias in AI-based systems, including blind acceptance and lack of attention, and learn testing strategies that evaluate both the quality of recommendations and how users respond.

Document AI components to reveal design decisions, usage, and testing needs. Include identifiers, hardware, data sources, biases, thresholds, privacy, and performance metrics.

Explore concept drift in real-world AI systems, learn continuous monitoring, retraining with updated data, and validation through confirmation, regression, and A/B testing to ensure robust performance.

Apply risk-based testing to ML systems by blending traditional and AI-specific approaches, focusing on data quality, pipelines, model development, performance metrics, and user expectations to manage uncertainty.

Develop tests that verify autonomous AI systems correctly decide when to operate independently and when to request human intervention, using boundary value analysis and varied environmental scenarios.

Explore testing probabilistic and non-deterministic AI systems, where outputs vary and exact values are unpredictable; apply tolerances, behavioral validation, repeated tests, and black-box analysis of interacting components.

Improve testing of AI systems by mastering transparency, interpretability, and explainability; learn how information availability, model behavior, and user perspective shape validation, testing methods, and trust.

Explore the test oracle concept in AI-based systems, explain ground truth and the oracle problem, and learn alternative techniques like metamorphic, A-B, and back-to-back testing.

Drives risk-based test objectives for AI systems by blending classic software testing with AI concerns such as bias, explainability, adaptability, concept drift, and ethics, ensuring performance, reliability, security, and autonomy.

Explore testing for algorithmic, sample, and inappropriate bias across the ML lifecycle, using independent data sets and techniques like LIME to ensure fair, balanced, and robust AI systems.

Explore pairwise testing, a combinatorial technique that selects a subset of all possible combinations to cover every pair of parameter values in AI-based systems and detect defects efficiently.

Learn back-to-back testing, or differential testing, to validate AI systems by comparing outputs from two independent systems using a pseudo-oracle when no clear expected result exists.

Apply A/B testing to compare two variants, version A and version B, using measurable outcomes from real-world inputs to inform data-driven decisions in AI systems and production environments.

Learn metamorphic testing (mt) and metamorphic relations to create follow-up test cases and verify output relationships, enabling defect detection in AI systems without exact results.

Experience-based testing in AI systems leverages tester intuition with error-guessing, exploratory testing, and checklists to identify bias, data issues, and model risks, complementing formal methods for a balanced strategy.

Select testing techniques for AI-based systems, balancing AI and non-AI components by risk and system characteristics, using back-to-back, metamorphic, A-slash-B, adversarial, and pairwise testing to mitigate the oracle problem.