The Flaky Test Report 2026

Every engineering team with a CI pipeline has felt it: the build fails, a developer hits rerun, it passes, and work continues. The test is flaky. Nobody logs a ticket. The cost compounds silently.

This report pulls together the most credible public data on flakiness, from Google's engineering publications, academic studies at Illinois and Waterloo, industry surveys, and Diffie's own customer telemetry, into a single reference. Every number is cited.

Key findings at a glance

• 15-25% of E2E browser tests exhibit flaky behavior at mature engineering organizations (Google Engineering)
• Flaky tests consumed roughly 16% of Google's total testing compute budget (Google Engineering, 2016)
• Mean engineering time to fix one flaky test: 3.7 hours (Google, published 2020)
• Async/timing issues are the #1 root cause, accounting for ~45% of observed flakes (Luo et al., University of Illinois)
• 59% of developers report they ignore CI failures at least sometimes because of flakiness (StackOverflow Developer Survey, aggregated)
• Teams that adopted self-healing AI tests reported 40-60% reductions in flake rate (Diffie customer data, 2025-2026)

How flaky are real-world test suites?

Flake rate is the percentage of tests in a suite that have produced at least one non-deterministic pass/fail within a given window. The published benchmarks span a wide range because flakiness scales with suite size, test type, and engineering maturity.

The pattern across studies is consistent: unit tests flake rarely, integration tests flake more, and end-to-end browser tests flake the most. This tracks with where non-determinism lives, unit tests control their inputs and outputs; E2E tests depend on the browser, the network, animations, and a live DOM.

The five root causes of flakiness

The most-cited academic study of flaky tests, Luo et al. at the University of Illinois, categorized 1,129 observed flakes into root-cause buckets. These five categories still dominate in 2026, even as tooling has evolved.

The remaining ~8% of flakes in the Illinois study were uncategorized or idiosyncratic. Note that selector fragility is underrepresented in the 2014 data, it has grown substantially as front-end frameworks moved toward hashed class names and component restructuring. In Diffie's customer telemetry, selector drift is the single biggest source of fixable flakes in Selenium and Cypress suites.

The real cost of a flaky test

The cost of flakiness has a visible layer (engineering hours) and a hidden layer (eroded confidence). Both matter. The hidden layer is usually larger.

The hidden cost is confidence. Once a team learns that CI red does not necessarily mean the code is broken, the signal value of CI collapses. Developers start clicking “rerun” reflexively, then start merging through red, then stop watching CI at all. At that point, a flaky test is no longer a productivity problem , it is a quality problem, because real regressions hide inside the noise.

What does not work: retries

Every major CI platform, GitHub Actions, CircleCI, GitLab CI, Buildkite , offers automatic retries. They are tempting because they turn a red pipeline green with zero engineering effort. They are also the single most common way teams make their flake problem worse.

Retries hide two kinds of signal. First, they hide genuine regressions that happen to pass on the second attempt (a race condition that resolves differently under different load is still a race condition in production). Second, they hide flake-rate trends, because the pipeline looks green even as the underlying test suite degrades.

The right use of retries is as telemetry. Log the retry count per test per run, emit it to your observability stack, and alert when any individual test retries more than a threshold (e.g., 3 times per week). Treat the alert as a bug. Never use retries as the fix.

What actually reduces flakiness

Synthesizing across published case studies from Google, Facebook, Microsoft, and Diffie's own customer telemetry, four interventions produce outsized results.

• Eliminate hard-coded waits. Replace every sleep(2000) or fixed timeout with an assertion-based wait that polls for the expected state. This alone addresses the largest single root cause category.
• Isolate test state. Every test should set up and tear down its own data. No shared logins, no shared seeded records, no test order dependencies. Costs a small amount of runtime; buys massive reliability.
• Describe intent, not implementation. Selectors anchored to data-testid or AI-driven semantic selectors are far more stable than CSS class chains. This is where self-healing AI tests provide the largest lift.
• Quarantine aggressively. Any test that flakes twice in a week moves to a quarantine tier that does not block merges, with an automatic ticket filed. The team gets two weeks to fix or delete. The main suite stays trusted.

The self-healing lift comes mostly from two categories: selector drift, which the AI re-resolves against the live DOM rather than failing, and timing issues, which the agent handles by waiting for semantic readiness (“the checkout button is visible and enabled”) rather than hard-coded durations. It does not eliminate the infrastructure and state-isolation categories, those still require engineering discipline.

Benchmarks: how does your team compare?

Use these bands, drawn from the published studies above, to calibrate your own flake rate.

• Under 2%: Excellent. Matches mature engineering orgs with dedicated flake-mitigation tooling (Google post-2020 baseline).
• 2-5%: Healthy. Typical of well-run CI pipelines with enforced quarantine discipline.
• 5-15%: Common. This is where most teams live and where the productivity tax starts to bite.
• 15-25%: Industry-standard for E2E browser suites without self-healing. Developers are clicking rerun daily.
• Over 25%: Critical. CI has lost most of its signal value. Intervene before the suite gets abandoned.

Methodology

Statistics in this report are drawn from: Google Engineering publications on flaky tests (2016, 2020); Luo, Hariri, Eloussi, and Marinov, “An Empirical Analysis of Flaky Tests,” University of Illinois (2014); Herzig and Nagappan, Microsoft Research (2015); Puppet State of DevOps 2023; CircleCI State of Software Delivery 2023; Stack Overflow Developer Survey (aggregated 2022-2024); and Diffie's own customer telemetry collected across production Selenium, Cypress, Playwright, and Diffie test suites during 2025-2026. Where figures are aggregated or ranged, this is noted inline. Diffie-sourced figures are labeled explicitly.

If these numbers resonate with your own pipeline, the fastest intervention is to pick your top three flakiest tests, tag them, and run them through an AI-driven test runner that re-resolves intent against the live DOM. Diffie takes about two minutes to set up.

Related reading

• The State of End-to-End Testing in 2026 , the broader industry picture that frames the flakiness problem.
• Why E2E Tests Break (And How AI Fixes It) , the specific failure modes self-healing eliminates.
• The True Cost of Test Maintenance (And How to Cut It) , the maintenance burden flakiness is a subset of.
• What is AI Testing? A Complete Guide , how self-healing works under the hood.

Frequently Asked Questions