You inherited a codebase with little automated testing, added checks around recent changes, and saw the test count rise. That does not tell you whether releases are safer. I would judge the QA effort by whether it finds consequential defects before release without making delivery unworkable. That requires a release-level baseline, a way to test the checks themselves, and honest treatment of failures you never observe.
A release ledger is more useful than a test-count target
Start with the unit you are trying to make safer: a production change. Create one record per deployed change, including its release time, affected component, change type, risk assessment, tests run, reviewer, and whether it was later linked to a production defect. GitHub Actions can supply test results and deployment IDs; Jira can hold the incident link; Sentry can help establish when a defect first appeared. None of those tools should silently decide causation, because a report opened after a release may describe a bug that was already present.
Define an “escape” before inspecting the results. One workable definition is a customer-visible P1 or P2 defect first introduced by a change and confirmed within 14 days of its deployment. Treat that window as a starting value to tune: a billing job that runs monthly needs longer observation than an endpoint exercised every hour. Record the date of introduction separately from the date of discovery where evidence permits. Otherwise, faster detection can make the QA effort appear to increase defects even while it reduces customer exposure.
I would not use the percentage of lines covered as the success measure, because coverage.py with branch = True can show that a branch ran without showing that any assertion would catch its wrong outcome. Nor would I set a target for the number of tests: a thousand checks of stable formatting can overwhelm a useful check of a risky state transition. Treat Test Inherited Code Cheaply Without Faking Assurance as a budget constraint to examine, not a result to declare; cheap tests earn their place when their cost buys observable detection.
Keep a compact set of measures beside the escape rate. Track P1/P2 escapes per deployed change, the proportion of releases with a rollback, time from first customer impact to detection, and the hours spent maintaining or investigating tests. Separate component-level results: an overall improvement can hide deterioration in the one component receiving most risky changes. Record release frequency and change size as context, since fewer, larger releases can lower the apparent number of escapes per release without lowering the risk carried by each change.
For a simple ledger, save one row per deployed change in change_events.csv, with columns cohort, released_at, and escaped_p1_p2_14d. Run this Python 3 script as python3 escape_rate.py change_events.csv:
import csv, datetime as dt, sys
from collections import defaultdict
today = dt.date.today()
totals = defaultdict(lambda: [0, 0])
with open(sys.argv[1], newline="") as file:
for row in csv.DictReader(file):
age = (today - dt.date.fromisoformat(row["released_at"])).days
if age >= 14:
group = row["cohort"]
totals[group][0] += int(row["escaped_p1_p2_14d"])
totals[group][1] += 1
for group, (escapes, changes) in sorted(totals.items()):
print(group, changes, escapes, round(100 * escapes / changes, 1) if changes else "n/a")
The script excludes releases too recent to have completed the window, because counting them as clean would favor the newest cohort. It counts changes with an escape, not the number of incidents; retain incident IDs elsewhere so one release that causes several failures is still visible in an incident review.
A falling escape rate is not proof until the comparison is fair
Inherited systems rarely offer a clean before-and-after experiment. The team may deploy less often while adding tests, or move from a volatile subsystem to a stable one. Compare similar changes in similar components over equal observation windows, and annotate major changes in traffic, instrumentation, and incident-reporting practice. If OpenTelemetry tracing or Sentry reporting begins halfway through the comparison, earlier incidents may simply have been harder to find.
For example, suppose a worked ledger shows 7 escapes among 80 eligible changes before the intervention and 3 among 75 afterward. Those are illustrative observations, not measurements from your system. The rates are about 8.8% and 4.0%, but so few events cannot support a confident claim that testing caused the difference. Inspect the affected components and change sizes, then keep collecting releases rather than announcing a percentage improvement. Report the counts alongside the rates so nobody mistakes a small denominator for precision.
A stronger comparison is possible when two groups have equally plausible access to the new checks. For a limited period, assign eligible low-risk changes within the same component to either the existing review path or the new test-assisted path, then apply the same release and incident criteria to both. Do not withhold a known safety check from a high-risk fix merely to preserve the comparison; the experiment would impose an avoidable cost. If assignment is impractical, match changes by component, change type, and size, and describe the result as observational rather than causal.
Escapes also need an exposure measure. A route called 2 million times after release has more opportunities to fail than an administrative task called twice; those call volumes are example values, not assumed traffic in your codebase. Prometheus counters and Grafana dashboards can supply request or job-run counts, while an OpenTelemetry trace_id can connect a failure to a deployed version. Compare defects per meaningful operation where operations are comparable, but retain per-change counts too: traffic-normalized rates alone can make a rarely used, critical path disappear.
Give the team a rule for interpreting ambiguous cases. Review each potential escape with a developer and an operations or support representative; record “introduced,” “pre-existing,” or “unknown,” plus the evidence. Keep unknowns in the report instead of forcing them into the favorable category. Use Inherited an untested codebase What QA advice is obsolete now to challenge habitual QA targets, then require the replacement targets to survive this classification exercise. A metric that depends on relabeling disputed incidents is not measuring safer releases.
The checks must demonstrate that they can detect a failure
Escapes are a lagging measure, and a quiet month can occur by chance. Test the detection system directly with defects the team already understands. Take a sample of previously fixed production bugs, reintroduce each fault on an isolated branch, and run the checks that would have run for the original change. Record whether a test fails for the intended reason. This is not proof about every future bug, but it tests a concrete claim: that the current checks recognize failure modes the codebase has actually produced.
Use mutation testing for smaller, well-defined areas. PIT can change Java expressions and report surviving mutants; StrykerJS can do similar work in JavaScript or TypeScript. A surviving mutant is a prompt to inspect the behavior, not automatically a demand for another assertion, because some mutations are equivalent to the original behavior or fall outside the path under study. Pair mutation results with Playwright checks for user-visible flows and Pact contract checks where an external interface is the failure boundary. Name the behavior each check is intended to detect; tool output alone cannot tell you whether the behavior matters.
Set an initial sampling rule you can afford to follow. For instance, make 10 historical defects an initial sample size to tune, chosen across affected components and failure types rather than from whichever bug has the easiest fixture. If the checks catch only the input-validation bugs while missing state or timing failures, the result tells you where the blind spot lies. Keep a separate count of bugs that cannot be reproduced reliably; excluding them without explanation would make the checks look stronger than they are.
Measure detection at the moment it could have mattered. A Playwright test added after an incident may catch its reproduction today yet have offered no warning on the original release. Record when the check entered GitHub Actions, whether it ran on the relevant pull request, and whether a failing result would have blocked deployment. Also review false alarms: if a flaky check fails frequently and engineers routinely rerun it without investigation, its apparent sensitivity may have little operational value.
Before accepting a new check, ask a falsifiable question: “Which plausible wrong behavior should make this fail?” Demonstrate that failure locally when feasible, then retain the evidence in the pull request. This practice costs more than merely asserting that a page loads, which is why it belongs first on paths implicated by incidents or upcoming risky changes. It turns test additions into measured detection attempts rather than an accumulation of green results.
Canaries beat replay for live integration risk, while replay wins on reproducibility
Two options deserve an explicit choice. Offline replay runs recorded requests or events against a candidate build and compares selected outputs; it wins when inputs can be captured safely and deterministic differences matter, but it costs storage, sanitization work, and maintenance whenever schemas change. Production canary sends a small share of real traffic to the candidate; it wins when failures depend on live integrations or current data, but it costs routing and rollback machinery and exposes some users to a faulty build. Neither option is a universal confidence score, because each observes a different slice of failure.
For a canary, define the stop rule before deployment. A proposed 5% traffic share is a setting to tune, not evidence that the risk is small: a rare operation might receive no canary traffic at all. Compare canary and established versions on the same Prometheus error counters and latency histograms, and check that the versions see comparable request types. A lower canary error rate means little if routing sends its hardest requests elsewhere. Record both automatic rollbacks and cases where a human overrode the alarm, with the reason for the override.
For replay, define which differences are meaningful. Timestamps, generated IDs, and unordered collections can change without a defect, while a changed authorization decision should not be normalized away. Save examples of accepted and rejected differences and audit them after a schema change. If replay catches discrepancies but most are ignored as noise, measure the investigation time as well as the catch count; a noisy detector can slow reviews until its useful alerts are also dismissed.
Whichever option you choose, watch a balancing measure: elapsed time from a change being ready to its production deployment. Slower delivery may be an acceptable price for fewer serious escapes, but it should be a deliberate trade rather than a hidden effect of flaky checks or manual sign-off queues. A test strategy is working when its detection evidence and release outcomes improve enough to justify that price, not when its dashboard has the most green indicators.
The first useful result is a baseline you can dispute
Tomorrow, take the last completed set of deployed changes and build the release ledger with one developer and one incident responder. Agree on the escape definition, classify disputed defects as unknown, and mark which changes have completed the observation window. Then pick one recurring failure mode and test whether today’s checks would catch a reintroduced example. That gives you a baseline and a detection experiment before you ask the team to trust another QA target.
