AI Test Generation: How It Actually Works

“AI writes your tests” sounds like magic, and marketing departments have leaned into that. But there is no magic. AI test generation is a concrete technical process with specific steps, real tradeoffs, and known limitations. Understanding how it works helps you decide where it fits and where it does not.

This post walks through the pipeline from the moment you type a test description to the moment you get a result. We will use Diffie as the reference implementation, but the core concepts apply broadly to AI testing tools.

Step 1: Natural language in, structured intent out

You type something like: “Go to the pricing page, switch to annual billing, and verify the Business plan shows $79/month.”

This sentence is not executed directly. An LLM (large language model) parses it and extracts a sequence of intended actions: navigate to a URL, find and interact with a billing toggle, locate a specific plan card, and check a price value. The model understands that “switch to annual billing” implies clicking a toggle or tab, not typing text into a field.

This step is where the AI differs from a recorder or a codegen tool. A recorder captures exactly what you did (click at coordinates X,Y). A codegen tool translates your recording into framework code. The AI understands what you meant, which means it can adapt when the page looks different than expected.

Step 2: A real browser launches

AI test generation is not simulation. A real Chromium browser instance launches in a cloud environment. The AI agent controls this browser through an automation protocol (typically the Chrome DevTools Protocol or a library like Playwright that wraps it).

The browser navigates to your application. The page loads with real JavaScript, real API calls, real CSS rendering. There is nothing synthetic about this step. If your app has a bug, the AI will encounter it the same way a user would.

This is an important distinction from some “AI testing” approaches that analyze source code or DOM snapshots without running the application. Those approaches can find potential issues but cannot verify that the application actually works. Browser-based AI testing verifies behavior in a live, running application.

Step 3: See the page, decide the action

Here is where AI test generation fundamentally diverges from traditional automation. In Selenium or Playwright, the script says: “find element with selector #billing-toggle and click it.” If that selector does not exist, the test fails.

An AI agent takes a different approach. It observes the current state of the page — the visible elements, their text, their layout, their apparent purpose — and decides what to interact with based on the intent of the current step. “Switch to annual billing” might mean clicking a toggle labeled “Annual,” selecting a radio button, or clicking a tab. The AI figures out which one is present on this specific page.

The observation can happen through multiple channels. The AI might receive a structured snapshot of the page's accessibility tree (the same tree screen readers use), a screenshot that it interprets visually, or a combination of both. The accessibility tree is particularly useful because it provides semantic meaning: this element is a button, this is a text input, this is a navigation link.

Step 4: Execute, observe, repeat

The AI does not plan all actions upfront and execute them in sequence. It works in a loop: observe the page, decide the next action, execute it, observe the result, decide the next action. This loop continues until the test intent is fulfilled or the agent determines it cannot proceed.

This iterative approach is what makes AI tests resilient. If clicking the annual toggle triggers an animation and the price takes 500 milliseconds to update, the AI observes that the page is still changing and waits. No explicit timeout, no hardcoded sleep, no retry logic. The agent sees that the page has not yet reached the expected state and continues observing.

If something unexpected happens — a cookie banner appears, a modal pops up, the page redirects — the AI adapts. It dismisses the banner, closes the modal, or follows the redirect, then resumes the original task. A traditional test script would fail on the unexpected element because it was not part of the scripted sequence.

Step 5: Assertions without selectors

The final part of most tests is verification: did the right thing happen? In traditional frameworks, assertions look like expect(element.text()).toBe('$79/mo'). The assertion is tied to a specific element found by a specific selector.

AI assertions work at the intent level. “Verify the Business plan shows $79/month” means: find the section of the page that represents the Business plan, locate the price information within it, and check that it says $79/month (or $79/mo, or $79 per month — the AI understands these are equivalent).

This flexibility is a strength and a tradeoff. The AI correctly handles cosmetic variations that would break string-comparison assertions. But it also means the assertion is slightly fuzzy. If you need to verify that a value is exactly “$79.00” and not “$79” or “$79/mo,” an AI assertion may not catch that distinction. For most E2E testing, intent-level assertions are what you actually want. For financial precision testing, you may need more specificity.

What makes it “self-healing”

“Self-healing tests” is a marketing term that gets used loosely. Here is what it means technically in the context of AI test generation.

A traditional test breaks when the page changes because the test is bound to specific implementation details: a CSS class, an ID, an XPath, a DOM structure. Change any of those, and the test fails — even if the feature works perfectly.

An AI test is bound to intent, not implementation. “Click the submit button” works whether the button has class “btn-primary” or “submit-cta” or no class at all. It works whether the button is inside a form or inside a modal. It works after a complete UI redesign, as long as there is still a recognizable submit action on the page.

The test does not “heal” in the sense that something was broken and got fixed. Nothing was ever broken. The test description (“click the submit button”) is still accurate, and the AI can still fulfill it. Self-healing is really the absence of the fragility that required healing in the first place.

Where AI test generation falls short

Understanding the limitations is as important as understanding the capabilities.

• Network-level testing. AI agents interact with the rendered page, not the network layer. If you need to verify that a specific API was called with specific parameters, or mock an API to return an error, you need a traditional framework with network interception.
• Precise timing measurements. An AI agent can tell you that a page loaded, but not that it loaded in 230 milliseconds. Performance testing requires instrumentation that AI testing does not provide.
• State that is not visible. AI tests verify what a user can see and interact with. If a bug causes incorrect data to be saved to a database but the UI shows the right thing, an AI test will not catch it. You need API or database-level tests for that.
• Complex test data setup. Tests that require seeding specific database records, manipulating local storage, or configuring feature flags before the test runs are harder to express in natural language. Traditional frameworks handle this with setup hooks and fixtures.
• Cross-browser rendering differences. Most AI testing tools run on Chromium. If you need to verify that your layout works identically in Firefox and Safari, you need a tool with multi-browser support.

The right mental model

AI test generation is not a replacement for all testing. It is a replacement for the specific category of testing where a human clicks through the application to verify that things work. Login flows, checkout processes, form submissions, navigation paths, settings changes — the tests that verify your application does what your users expect.

Unit tests verify that individual functions return correct values. Integration tests verify that services communicate correctly. AI-generated E2E tests verify that the assembled application works from a user's perspective. They occupy the top of the testing pyramid — fewer in number, broader in scope, and traditionally the most expensive to write and maintain.

AI test generation makes the top of the pyramid dramatically cheaper. A test that took an hour to write and 20 minutes per month to maintain now takes two minutes to describe and zero minutes to maintain. That changes the economics enough that teams can afford more E2E coverage than before — not replacing the pyramid, but making its most expensive layer accessible.

Frequently Asked Questions