How to Test AI Applications: QA Strategy, Methods & Checklist

An AI application can return the correct answer 95 times out of 100 and still be unsafe to release. The remaining five responses may disclose private data, disadvantage a specific user group, invent medical information, execute an unauthorized action, or fail in a way that conventional functional tests never examine.

That is why testing an AI product cannot stop at checking whether the model is “accurate.” QA must validate the data, model, prompts, retrieval pipeline, integrations, security controls, user experience, and production behavior as one connected system.

The need for this broader approach is becoming more visible. Stanford’s 2026 AI Index recorded 362 documented AI incidents in 2025, up from 233 in 2024. It also reported hallucination rates ranging from 22% to 94% across 26 leading models on one accuracy benchmark. Under adversarial prompting, safety performance declined for every tested frontier model. The findings are summarized in Stanford’s latest responsible AI research.

This guide explains how to test AI applications systematically, which metrics to use, how to evaluate generative and agentic systems, and what must be monitored after release.

What is AI application testing?

AI application testing is the process of evaluating whether an AI-enabled product performs its intended task reliably, safely, fairly, securely, and within defined operational limits.

The object under test is not only the model. A production AI system usually includes:

• Input validation and data-processing pipelines;

• A machine-learning or foundation model;

• System and user prompts;

• Retrieval-augmented generation components;

• Vector databases and knowledge sources;

• APIs and external services;

• Business rules and deterministic code;

• Tool-calling or agent functionality;

• Moderation and security controls;

• User interfaces;

• Logging and monitoring systems;

• Human-review workflows.

A model can perform well in isolation while the complete product fails. For example, a support assistant may generate accurate answers during offline evaluation but retrieve an outdated policy in production, expose information from another customer’s document, or send an unauthorized refund request through an integrated tool.

Testing must therefore cover both model quality and system behavior.

This topic is different from using AI to generate tests or maintain automation. For that use case, see our guide to AI in software testing.

Why testing AI is different from traditional software testing

Traditional software is generally expected to produce a deterministic result for a defined input. AI systems operate probabilistically. The same request can produce different responses, and several outputs may all be acceptable.

This changes the testing problem.

The 2026 International AI Safety Report describes an “evaluation gap”: strong performance in pre-deployment tests does not reliably predict real-world utility or risk. It also notes that advanced models can excel at difficult tasks while failing at apparently simpler ones, and that some systems increasingly recognize evaluation settings or exploit loopholes. The report’s independent expert assessment reinforces the need to test realistic workflows rather than rely on model benchmarks alone.

What should be tested in an AI system?

A complete AI testing strategy covers multiple layers.

Not every application requires the same depth. A movie recommendation tool and a system supporting medical decisions should not share identical release criteria. Test intensity must reflect potential harm, regulatory exposure, reversibility, and the degree of autonomy granted to the system.

Step-by-step process for testing AI applications

1. Define the intended use and unacceptable outcomes

Start with the decision or task the AI system is allowed to perform.

Document:

• Intended users;

• Supported languages, regions, and devices;

• Permitted inputs;

• Expected outputs;

• Unsupported use cases;

• Required human oversight;

• Maximum acceptable latency and cost;

• Known limitations;

• Potentially affected groups;

• Consequences of an incorrect result;

• Actions the system may perform autonomously;

• Actions that always require approval.

Then define unacceptable outcomes.

For a financial assistant, these might include inventing account balances, exposing transaction details, giving personalized investment advice without authorization, or executing a transfer without confirmation.

For a hiring model, unacceptable behavior might include using protected characteristics, applying inconsistent criteria to equivalent candidates, or generating recommendations without an auditable explanation.

NIST’s 2026 draft TEVV-Athlon framework recommends tailoring test, evaluation, verification, and validation activities to the organization’s specific objectives, system, context, and potential impact. Its AI system evaluation framework treats evaluation as evidence that a system can meet its goals while limiting negative consequences—not as a generic benchmark exercise.

2. Convert risks into testable requirements

“Must be fair,” “must not hallucinate,” and “must be safe” are goals, not test cases. Convert them into observable requirements.

Instead of:

The assistant must not reveal sensitive information.

Define:

When a user asks for another customer’s order, contact details, payment data, or support history, the assistant must refuse, reveal no protected fields, and create no unauthorized tool call.

Instead of:

The recommendation model must be fair.

Define:

The false-negative rate must be measured for each relevant demographic group, and any difference beyond the approved tolerance must block release or require a documented risk decision.

Each requirement should have:

• A measurable outcome;

• An evaluation dataset;

• A pass/fail threshold or review rubric;

• An owner;

• A severity if the requirement fails;

• Evidence retained for audit;

• A rule for escalation or release blocking.

3. Test the data

In AI systems, data influences behavior as directly as application code. Data testing should cover training, validation, evaluation, retrieval, and production inputs.

Data quality checks

Validate:

• Schema and types;

• Missing and null values;

• Duplicate records;

• Invalid labels;

• Unexpected ranges;

• Broken relationships;

• Encoding and language issues;

• Outliers;

• Timestamp consistency;

• Data freshness;

• Transformation logic.

Representativeness

The dataset should cover the conditions in which the system will operate:

• Relevant demographic groups;

• Languages and dialects;

• Geographic regions;

• Devices and channels;

• Common and rare cases;

• Seasonal behavior;

• Noisy and incomplete inputs;

• Edge cases;

• Adversarial inputs.

Overall accuracy can hide poor performance for smaller groups. Evaluate important slices independently and compare error rates, not only average scores.

Google’s fairness research recommends sliced analysis and counterfactual testing because summary accuracy does not describe how different groups experience a product. Its discussion of fairness measurement in production systems also highlights that fairness, accuracy, latency, and user satisfaction should be assessed together rather than treated as isolated objectives.

Leakage and contamination

Check whether evaluation data appeared in training or was derived from it. Leakage can make performance look substantially better than it will be in production.

For generative systems, contamination can occur when:

• Benchmark questions were included in model training;

• Evaluation examples appear in prompt templates;

• Retrieved documents contain expected answers;

• Human reviewers have access to reference labels during supposedly blind assessment;

• Test conversations are reused in fine-tuning and later evaluation.

Maintain separate datasets for development, tuning, validation, and final evaluation. Restrict access to the final holdout set where the risk justifies it.

4. Build a representative evaluation dataset

An AI test set should represent expected behavior and known risks, not merely contain random examples.

A useful evaluation set normally includes:

• Typical user requests;

• Business-critical scenarios;

• Boundary conditions;

• Rare but harmful failures;

• Ambiguous inputs;

• Invalid inputs;

• Multilingual examples;

• Long and short context;

• Conflicting instructions;

• Adversarial prompts;

• Previously reported production failures;

• Requests the system should refuse;

• Cases requiring human escalation.

Store each example with metadata such as risk category, language, user group, expected behavior, source, and severity. This allows results to be analyzed by slice instead of collapsing everything into one average.

For generative applications, define a scoring rubric. A customer-support response might be judged on factual correctness, policy compliance, completeness, tone, citation quality, and whether escalation was appropriate.

5. Select metrics that match the task

There is no single “AI accuracy” metric. The correct metrics depend on what the system does and the cost of different errors.

Classification metrics

A fraud model can achieve high accuracy by marking nearly everything legitimate if fraud is rare. Recall, precision, financial impact, and threshold-specific behavior provide more useful evidence.

Regression metrics

Use measures such as mean absolute error, root mean squared error, median error, percentile error, and performance within business-defined tolerance bands.

Do not report only an average. Inspect the worst errors and important data segments.

Generative AI and LLM metrics

Automated LLM evaluators can accelerate scoring, but they introduce another probabilistic model into the process. Calibrate automated evaluation against qualified human reviewers and periodically test agreement.

Retrieval-augmented generation metrics

For RAG systems, test retrieval separately from generation.

Useful measurements include:

• Retrieval precision;

• Retrieval recall;

• Ranking quality;

• Document freshness;

• Access-control accuracy;

• Answer groundedness;

• Citation correctness;

• Unsupported-answer rate;

• Correct abstention when no evidence exists.

A generator cannot reliably answer from the correct source if the retrieval layer provides the wrong document.

Agentic AI metrics

Agent testing should measure both goal completion and the path taken.

Track:

• Goal-completion rate;

• Valid tool selection;

• Tool-call parameter accuracy;

• Unauthorized-action rate;

• Policy violations;

• Unnecessary actions;

• Recovery after a failed step;

• Human-intervention frequency;

• Time and cost per completed workflow;

• Ability to stop when instructed;

• Audit-trail completeness;

• Results across repeated executions.

A successful final result does not justify an unsafe route. An agent that completes a refund after accessing an unrelated customer account should fail the test even if the refund itself is correct.

6. Perform functional and system testing

The deterministic parts of an AI product still require conventional QA.

Test:

• Authentication and authorization;

• Input validation;

• API contracts;

• Database updates;

• Session management;

• File processing;

• Error handling;

• Retry and timeout behavior;

• UI workflows;

• Browser and device compatibility;

• Accessibility;

• Third-party integrations;

• Logging and audit records.

Then validate the model inside complete user journeys.

An AI travel assistant may provide a plausible itinerary in isolation, but the full application must also handle unavailable flights, stale prices, currency conversion, API failures, user preferences, booking permissions, and confirmation steps.

Model evaluation and software testing should meet at the system boundary.

7. Test fairness and harmful bias

Fairness testing begins by identifying who may be affected and how an incorrect decision could harm them.

The process should include:

Counterfactual testing is particularly valuable. If two inputs are equivalent except for a protected or potentially sensitive characteristic, the output should differ only when there is a legitimate, documented reason.

Fairness does not always mean identical results. The correct definition depends on the decision, legal context, affected population, and harm being controlled.

8. Test robustness and edge cases

An AI system should be tested outside clean benchmark conditions.

Introduce:

• Misspellings and grammatical errors;

• Unusual formatting;

• Empty and extremely long inputs;

• Low-quality images or audio;

• Contradictory context;

• Irrelevant information;

• Changes in input distribution;

• Unseen categories;

• Rare combinations;

• Reordered or paraphrased requests;

• Delayed and failed dependencies.

Metamorphic testing is useful when an exact expected result is difficult to specify. Instead of asserting one output, define relationships that should remain true.

Examples:

• Paraphrasing a request should not materially change the answer.

• Reordering independent facts should not change a classification.

• Adding irrelevant information should not reverse a decision.

• Converting equivalent units should preserve the result.

• Changing a demographic attribute should not affect an unrelated recommendation.

Run each case more than once if the model is non-deterministic. Report distributions, not only a single response.

9. Conduct adversarial and security testing

AI applications introduce attack paths beyond traditional web and API vulnerabilities.

The 2025 OWASP Top 10 for LLM and generative AI applications includes prompt injection, sensitive-information disclosure, supply-chain weaknesses, data and model poisoning, improper output handling, excessive agency, system-prompt leakage, vector and embedding weaknesses, misinformation, and unbounded resource consumption. Each category in the OWASP GenAI risk framework should be mapped to relevant test scenarios.

Prompt injection

Test direct instructions such as attempts to override the system prompt, but also indirect injection hidden in:

• Retrieved web pages;

• Uploaded documents;

• Emails;

• Support tickets;

• Images;

• Tool responses;

• Database records.

The system should distinguish untrusted content from trusted instructions.

Sensitive-information disclosure

Attempt to extract:

• Personal data;

• Credentials and secrets;

• System prompts;

• Proprietary documents;

• Other users’ conversation history;

• Training examples;

• Internal identifiers;

• Hidden chain-of-thought or implementation details.

Authorization must be enforced outside the model. A prompt telling the model not to reveal data is not an access-control mechanism.

Improper output handling

Treat model output as untrusted input. Test whether generated content can cause:

• Cross-site scripting;

• SQL or command injection;

• Unsafe HTML rendering;

• Invalid API calls;

• Path traversal;

• Malicious URLs;

• Code execution.

Excessive agency

Give agents the minimum tools and permissions required. Test whether they can:

• Act outside the user’s request;

• Bypass confirmation;

• Repeat financial or destructive operations;

• Access another tenant;

• Call an unapproved tool;

• Continue after authorization is revoked;

• Follow malicious instructions returned by a tool.

Resource abuse

Test extremely long inputs, recursive workflows, repeated tool calls, expensive queries, and conversations designed to maximize token consumption. Define hard limits on time, tokens, cost, steps, and retries.

10. Test explainability and human oversight

An explanation should help an appropriate user understand and challenge a result. It should not merely sound persuasive.

Test whether:

• The system communicates uncertainty;

• Explanations reflect the actual evidence used;

• Important limitations are disclosed;

• Users know they are interacting with AI;

• Users can request human review;

• Operators can trace inputs, model version, retrieval sources, and actions;

• A reviewer can override or stop the system;

• Explanations remain understandable to the intended audience.

High confidence should not be treated as proof of correctness. Validate whether confidence scores are calibrated against real outcomes.

11. Test performance, reliability, and cost

AI applications can pass quality evaluations and still fail operationally.

Measure:

• Time to first response;

• Total response time;

• Throughput;

• Concurrent-user capacity;

• Timeout rate;

• External API failure rate;

• GPU, CPU, and memory consumption;

• Token consumption;

• Cost per request;

• Cost per successful task;

• Queue behavior;

• Rate-limit handling;

• Fallback performance;

• Recovery from provider outages.

Test quality under load as well as speed. A system may switch to a smaller model, truncate context, retrieve fewer documents, or increase timeouts during traffic spikes. Those changes can alter answer quality.

Include dependency failures:

• Primary model unavailable;

• Vector database unavailable;

• Retrieval results delayed;

• Tool call times out;

• Content filter fails;

• Logging service is unreachable;

• Context exceeds the permitted window.

The system should fail safely and tell users what happened instead of generating an unsupported answer.

12. Run human evaluation and usability testing

Some criteria cannot be reduced reliably to automated metrics.

Qualified reviewers should assess:

• Whether the answer is genuinely useful;

• Whether important context is missing;

• Whether tone is appropriate;

• Whether an explanation is understandable;

• Whether users over-trust the system;

• Whether escalation is clear;

• Whether the interface encourages unsafe actions;

• Whether errors can be recognized and corrected.

Use a documented rubric and train reviewers on examples. Measure agreement between reviewers. Low agreement may indicate that the requirement or rubric is too ambiguous.

Include users and domain specialists where mistakes have professional consequences. A general QA engineer can test interface behavior, but may not be qualified to judge a clinical recommendation, legal interpretation, or financial-risk decision.

13. Validate regulatory and governance requirements

Compliance depends on jurisdiction, industry, product purpose, and risk classification. It must be reviewed with qualified legal and compliance specialists.

For products operating in the EU, determine whether the system falls under the AI Act and which risk category applies. The European Commission states that high-risk systems are subject to obligations involving risk management, data quality, logging, documentation, human oversight, accuracy, robustness, and cybersecurity. Transparency requirements also apply to specified AI interactions and generated content. Current requirements and implementation dates are maintained on the Commission’s official AI Act overview.

Testing evidence may need to include:

• Risk assessments;

• Dataset documentation;

• Model and prompt versions;

• Evaluation results;

• Known limitations;

• Human-oversight controls;

• Incident records;

• Access logs;

• Change history;

• Post-market monitoring;

• Corrective actions.

Compliance should be built into the test strategy rather than assembled immediately before an audit.

14. Perform pre-deployment red teaming

Red teaming searches for harmful behavior beyond expected functional scenarios.

The red team should attempt to:

• Bypass safeguards;

• Manipulate system instructions;

• Extract protected information;

• Trigger harmful or discriminatory output;

• Misuse tools and integrations;

• Create conflicting goals;

• Exploit ambiguous authorization;

• Cause excessive resource use;

• Produce realistic misinformation;

• Circumvent human approval.

Include people who did not build the system. Independent testing reduces the risk that the evaluation reproduces the development team’s assumptions.

NIST’s Generative AI Profile recommends iterative and documented pre-deployment testing informed by representatives from across the AI lifecycle. The NIST GenAI risk profile also emphasizes third-party risks, including external models, datasets, services, and open-source components.

Red-team findings should become permanent regression tests wherever reproduction is safe and practical.

15. Monitor the AI system after release

AI testing continues in production because the operating environment changes.

Monitor:

• Input and feature drift;

• Output-distribution changes;

• Model quality on reviewed samples;

• Hallucination and refusal rates;

• Fairness metrics by relevant slice;

• Prompt-injection attempts;

• Unauthorized tool calls;

• User corrections and complaints;

• Human escalation;

• Latency and availability;

• Token and infrastructure cost;

• Changes in retrieval quality;

• Incidents associated with new model versions.

Define alerts, owners, and actions. An alert without a response process is only a notification.

Production feedback should feed a controlled evaluation loop:

AI testing techniques

Different methods reveal different failure modes.

No single method provides sufficient evidence. A reliable strategy combines offline evaluation, conventional software testing, adversarial assessment, human review, and production monitoring.

Example: testing an AI customer-support assistant

Consider a RAG-based assistant that answers questions using company policies and can create support tickets.

Functional tests

• Answers supported policy questions correctly;

• Asks for clarification when the request is ambiguous;

• Creates a ticket with the right category and priority;

• Preserves conversation context;

• Handles unavailable dependencies;

• Escalates requests outside its authority.

Retrieval tests

• Retrieves the latest policy;

• Does not retrieve another tenant’s documents;

• Ranks the correct source above irrelevant content;

• Provides citations that support the answer;

• Refuses to invent an answer when no source exists.

Safety and security tests

• Resists instructions hidden inside uploaded documents;

• Does not reveal system prompts or customer data;

• Sanitizes output before rendering;

• Cannot create unlimited tickets;

• Requires confirmation before sensitive actions;

• Stops tool execution when authorization expires.

Quality metrics

• Task-completion rate;

• Grounded-answer rate;

• Unsupported-claim rate;

• Correct-refusal rate;

• False-refusal rate;

• Citation accuracy;

• Escalation accuracy;

• Median and 95th-percentile latency;

• Cost per resolved request;

• Human-review agreement.

Production monitoring

• New unsupported questions;

• Policy changes;

• Retrieval failures;

• Customer corrections;

• Prompt-injection attempts;

• Increased escalation;

• Changes in latency or model cost.

The final release decision should be based on the entire system’s performance, not merely the model’s response score.

Common mistakes in AI application testing

Testing only the model

A strong model cannot compensate for broken permissions, stale retrieval, invalid API integration, or unsafe output handling.

Using one average score

Overall performance hides failures affecting rare cases, languages, or user groups. Always inspect critical slices and worst-case outcomes.

Treating an LLM judge as ground truth

Automated evaluation models can be biased, inconsistent, or sensitive to prompt design. Calibrate them against human review.

Reusing the same dataset indefinitely

The team gradually optimizes for the evaluation set, turning it into another training set. Keep protected holdouts and refresh cases from production behavior.

Ignoring negative and refusal scenarios

A system must be tested on what it should not do, not only on tasks it should complete.

Testing only in English

Language support must be verified separately. Translation does not preserve tone, ambiguity, cultural context, or performance automatically.

Giving agents broad permissions

Permissions should be enforced by application logic and infrastructure. Do not rely on the model to decide whether it is authorized.

Skipping post-release evaluation

New users, changing data, updated models, modified prompts, and third-party services can all change behavior without a conventional code release.

AI application testing checklist

Scope and governance

• Intended and prohibited uses are documented.

• Risk classification is agreed.

• Owners are assigned for model, data, security, QA, and compliance.

• Human-review and escalation rules are defined.

• Release-blocking failures are documented.

Data

• Schema, quality, and transformation checks are automated.

• Training and evaluation datasets are separated.

• Leakage and contamination have been assessed.

• Important groups and operating conditions are represented.

• Data lineage, consent, retention, and access are documented.

• Production drift will be monitored.

Model and output

• Metrics match the business consequences of errors.

• Results are measured by important slices.

• Confidence is calibrated where it affects decisions.

• Repeated runs are used for non-deterministic behavior.

• Edge, rare, and out-of-distribution cases are covered.

• Known limitations are documented.

Generative AI and RAG

• Factuality, relevance, completeness, and groundedness are evaluated.

• Retrieval is tested separately from generation.

• Citations are verified.

• Correct and false refusals are measured.

• Prompt-injection scenarios are included.

• Sensitive-information disclosure is tested.

• Unsupported answers trigger safe behavior.

Agents and integrations

• Tool permissions follow least privilege.

• Sensitive actions require confirmation.

• Tool parameters and outputs are validated.

• Step, time, token, and cost limits are enforced.

• Failed and malicious tool responses are tested.

• The agent can be stopped and rolled back.

• Every action is traceable.

Software quality

• Functional and integration tests cover the full workflow.

• APIs, UI, authentication, and authorization are tested.

• Performance and load limits are known.

• External-service failures have safe fallbacks.

• Accessibility and usability have been evaluated.

• Logging does not expose sensitive information.

Security, fairness, and compliance

• AI-specific threat modeling is complete.

• Adversarial and red-team testing has been performed.

• Fairness metrics and tolerances are approved.

• Privacy and data-protection requirements are verified.

• Regulatory obligations are mapped to test evidence.

• Third-party model and data risks are assessed.

Deployment and monitoring

• Model, prompt, data, and configuration versions are tracked.

• A baseline model or previous version is available for comparison.

• Canary, shadow, or rollback procedures are ready.

• Production-quality metrics have alerts and owners.

• Incidents become regression tests.

• Re-evaluation is triggered by relevant system changes.

A reliable AI test strategy evaluates the whole product

An AI solution is not reliable because it earned a high benchmark score. It is reliable when there is evidence that the complete product behaves acceptably across normal, unusual, adversarial, and changing conditions.

That evidence comes from several layers: representative data, task-specific metrics, conventional software testing, fairness analysis, security testing, human evaluation, controlled deployment, and continuous monitoring.

The higher the possible harm and the more authority the system receives, the stronger those layers must be.

DeviQA’s AI testing services help product and engineering teams build evaluation strategies, test AI-enabled workflows, validate security and performance, and establish production monitoring for AI systems.

A QA partner for every stage

Schedule a call

Frequently asked questions

How do you test an AI application?

Test the data, model, prompts, retrieval pipeline, application logic, integrations, security controls, user experience, and production monitoring. Use both quantitative evaluation and qualitative human review.

Can conventional QA methods be used for AI?

Yes. Functional, integration, API, security, performance, usability, compatibility, and end-to-end testing remain necessary. They should be combined with model evaluation, data validation, fairness testing, adversarial testing, and drift monitoring.

What is the most important metric for AI testing?

There is no universal metric. The most important measure is the one connected to the system’s intended outcome and the cost of failure. High-risk systems normally require several metrics and explicit limits for different user groups and failure types.

How do you test an AI model without a fixed expected result?

Use scoring rubrics, ranges, reference examples, invariant properties, metamorphic relations, pairwise comparison, domain-expert review, and repeated execution. The evaluation should define what makes an output acceptable even when there is no single correct wording.

What is the difference between model testing and AI system testing?

Model testing evaluates the predictive or generative component. AI system testing evaluates how the model behaves with prompts, data, retrieval, APIs, permissions, UI, users, and operational infrastructure. Both are required before release.

How often should an AI application be retested?

Retest after changes to the model, prompt, data pipeline, retrieval sources, business rules, tools, integrations, safety controls, or deployment environment. Continue monitoring in production and run scheduled evaluations for drift and emerging failure modes.

Can AI testing be fully automated?

Many checks can be automated, but complete evaluation usually cannot. Fairness, usefulness, ambiguous correctness, real-world harm, and high-impact decisions often require human and domain-expert judgment.

About the author

Anastasiia Sokolinska

Chief Operating Officer

Anastasiia Sokolinska is the Chief Operating Officer at DeviQA, responsible for operational strategy, delivery performance, and scaling QA services for complex software products.