QA
& Testing.

Inside this guide you'll find six real ways a passing suite told us something untrue, the check that finds tests which never ran at all, why a local verification can be a confident lie, how to test a payment path without spending anything, and the class of bug no automated test will ever catch.

Phase 1: What a Green Suite Does Not Mean

A green test run is a claim, and the claim is narrower than everyone treats it as. It says: the assertions that ran, passed. It does not say the assertions were correct, that they exercised anything, or that they all ran.

Three real cases, each from a codebase whose team believed it had good coverage.

The Three Statements a Suite Can Make

It helps to be explicit about which of these your suite is entitled to say, because teams routinely claim the third while owning only the first.

• "These assertions ran and passed." Almost every suite earns this. It is worth much less than it sounds, for the reasons in this phase.
• "These behaviours work." Earned only when the assertions exercise behaviour rather than structure, and when you have checked that they actually ran.
• "This product works." Never earned by a test suite alone. There is a class of defect, covered in Phase 7, that lives entirely outside what automated tests can observe.

Most testing arguments are two people making different statements from that list and assuming they are talking about the same thing.

Case Study: The Test That Encoded the Bug as Correct

An education platform's signup was completely broken. Every new parent account failed at the confirmation link — a technically valid, unexpired token that the client rejected as "invalid, expired, or already used."

The cause was a mismatch: the backend called an auth method that sends a signup-confirmation template, carrying the token in the URL fragment as one type; the frontend parser accepted only a different type in the query string.

There was a test for the parser. It existed, it ran on every commit, it passed, and it asserted that the wrong value was the correct one. The test had been written against the implementation rather than against the requirement, so it locked the bug in place and gave the team confidence about the exact code path that was down.

That is the most dangerous test in any suite: one that is green because it agrees with the bug.

Case Study: 28% of the Assertions Tested Nothing

The same codebase had 669 edge-function tests, all passing under full type-checking. Real tests, genuinely green.

Sampling and reading what they actually asserted, roughly 28% were readFileSync plus a regex over source text. The headline example: a test claiming to verify a 25-place cohort cap actually checked that the string FOR UPDATE appeared somewhere in a migration file. The migration was never executed. Nothing acquired a lock. Nothing tested concurrent access.

That test passes forever, including after someone deletes the row-locking behaviour and leaves the words in a comment.

Read what your assertions check, do not count them. A pass rate is a number about test files. Coverage of behaviour is a different thing and the two are only loosely related.

Case Study: 285 Out of 285, Actually 32 Failed

A handover document asserted "285/285 tests passing." A clean clone into a scratch directory, followed by npm test, produced 32 failed and 297 passed. Red.

Worse: a further 15 failures lived in four root scripts that npm test never invokes — including the safety-envelope guards, which are exactly the tests you would most want green on a product used by children. They were wired to separately named scripts nobody ran.

Two habits come out of this:

• A reported pass count is a claim, not a fact. Clone fresh, run it yourself, diff against the claim. Do not run it in the existing working copy — local state is precisely what you are trying to eliminate.
• List every script in the package file, not just the test one. Safety-critical suites hide behind names the default command never reaches.

Phase 2: The Test That Never Ran

A failing test is a signal. A suite that fails to load is silence, and silence reads as success in every summary line anyone actually looks at.

Case Study: Tests: 0 Total

On a marketing codebase, one component called an animation library's plugin registration at module scope — at import time, not inside a hook or an effect. The test environment has no matchMedia, and the animation library needs it during registration.

The result: any suite that so much as imported that component died with Test suite failed to run, and reported Tests: 0 total. Not one failure. Zero tests, on files that appeared to be covered, with a summary line that scrolls past looking unremarkable.

The fix was one global stub in the test setup file, alongside the existing observer and fetch stubs, guarded on the API not already existing.

How to Detect Zero-Coverage Suites

1. Grep your CI output for failed to run, not just for failed. They are different lines and only one of them is loud.
2. Assert a floor on total test count in CI. If the number drops below the expected count, fail the build. A suite that silently stops running is invisible without this.
3. Watch for module-scope side effects in components. Plugin registration, client construction, environment reads. Anything at module scope runs on import, which means it runs in every test that imports the file transitively.
4. Check coverage of files you believe are covered. A file with an import-time crash shows as untouched, which is easy to misread as "no test written yet."

Phase 3: Running the Right Tests

Case Study: One File, 184 Suites

Working in a worktree at ~/vds-openclaw-chrome, we ran the test runner filtered to a single feature name. It printed "Ran all test suites matching /openclaw-chrome/i" and ran all 184 suites, taking 85 seconds.

The reason is that the runner's path pattern matches the absolute path, and the pattern appeared in the worktree directory name. Every test in the repository matched.

The visible consequence is a slow run. The dangerous consequence is misreading: it looks like your one small change somehow broke thirty-nine unrelated suites, and that misreading can consume an hour.

Two fixes, both cheap:

• Use the runner's explicit single-file flag rather than a name pattern when you mean one file.
• Name worktree directories so they cannot collide with test filenames or feature names. A prefix is enough.

Phase 4: CI Gates That Do Not Gate

What Your Pipeline Actually Runs

Open the pipeline definition and read it. Not the badge, not the name of the job — the commands.

On a client repository, the aggregate CI command expands to type-check, lint, and test with coverage. Nothing in it builds the application. A build break is therefore completely invisible to CI and surfaces only on the hosting provider after merge. That is not a broken pipeline; it is a pipeline making a narrower claim than its name suggests, which is worse, because everyone reads the name.

The inverse case exists on the same repository: a local build fails on a generated-type constraint in a file untouched for seven months, while the deployed build — using a different bundler that does not enforce that constraint — succeeds on the same commit. So a local build failure there is not evidence of a break, and a passing CI run is not evidence the build works. Both directions require checking the preview deployment.

The Always-Red Gate

The same repository's static-analysis check has failed on every merged pull request for months. Everyone knows. Everyone merges anyway.

A gate that always fails provides exactly zero signal and trains the team to merge past red checks — which is the actual cost, because it generalises. The only two acceptable states are fixed or removed.

Phase 5: Verifying the Interface

Automated tests verify logic. Almost nothing in a test suite verifies that the thing a user sees is correct, which is why this phase exists and why it is mostly about not being lied to.

The Three Ways a Local Check Lies

One: the server answering is not the server you started. Kill a dev server, relaunch it in the background, sleep eight seconds, and curl. A 200 comes back. The page looks broken. In our case the new process had failed to bind because the old one was not dead yet, so the 200 came from a stale server serving the previous build, and every measurement taken afterwards described the wrong code entirely.

The fix is mechanical: kill by port, confirm the port is free, then poll until the new server answers, and only then measure. Never trust a fixed sleep to mean "the new server is the one responding."

Two: a narrow window is not a mobile viewport. A headless screenshot at a 430-pixel window width showed content clipped at the right edge — an obvious horizontal-overflow bug. Measured properly, documentElement.scrollWidth was exactly 430 and zero elements extended past it. The window-size flag sets the capture canvas but leaves desktop viewport semantics, so wide content is cropped rather than reflowed. A false positive, and one that would have sent a fix-hunt after nothing.

Real mobile verification means setting device metrics — width, height, scale factor, mobile flag — before navigating, then measuring, then capturing.

Three: the dev server itself can wedge. A branch's dev server stopped mounting a widget on the homepage while a second dev server on the main branch mounted it fine. It read exactly like a regression. Roughly fifteen minutes went into instrumenting the component, cold-clearing the build cache, and reverting hot-module updates before it was cleared — the deployed preview mounted it correctly on every route.

Two decisive tests, both faster than debugging, and both worth doing before you touch the component:

• Copy your changed files into a second worktree running the base branch's dev server. If the symptom does not follow the files, it is the server, not your code.
• Stash everything on the same server and reload. If the symptom persists on a clean tree, it is definitively the server.

Phase 6: Testing Money Without Spending It

Payment paths are the code most worth testing and the code teams test least, because the assumption is that testing it costs money. It does not.

Creating a checkout session charges nobody. A card is only taken when someone submits the hosted page. So a test can create the session, assert the returned URL and currency, even load the hosted page — and then abandon. That covers the entire portion of the flow that holds the money logic: line items, amounts, currency, metadata, success and cancel URLs.

Two practical extensions:

• When the pay button is gated behind app state you cannot easily reach, call the underlying function directly. One app we tested required ten profile views before the button appeared. Testing the gate is a separate test from testing the payment.
• Reproduce against the real host. Both zero-payment bugs we found on that engagement were live on the production domain, and both had been explained away as "we just haven't had many sales yet." One curl against the real host settled it in under a minute.

Pinning a Bug You Cannot Fix

Some defects are on the other side of a boundary you do not control — a platform's access rule, a vendor's API, another team's service.

Do not skip those tests. Mark them as expected-to-fail. The suite stays green while the defect exists, and the runner reports "expected to fail but passed" the moment the other side ships a fix — which is often the only notification you will get. A skipped test tells you nothing, forever, and quietly becomes permanent.

Phase 7: The Bugs Tests Structurally Cannot Catch

Case Study: The Double Charge

A real buyer paid twice for the same product, five minutes apart. His description: the button didn't change on press, so he clicked it again.

The mechanics: the pay button had no pressed state and was not disabled while the payment intent was in flight. Two payment intents were created — one for $473, one for $297 for the identical tier — and the fulfilment system minted two separate download records pointing at the same file, so nothing downstream flagged it either. He emailed twice within four minutes asking for a refund and reporting that he had received no files at all.

No unit test catches this. No integration test catches it. The logic is correct on both invocations; the bug is that a human, receiving no feedback, produced a second invocation. It is a bug in the gap between the interface and the code, and only a human clicking, or a test explicitly written to double-fire, finds it.

Which is the point of this phase. There is a class of defect that lives outside what test suites are shaped to observe, and every one of them is expensive:

• Double submission on any action that costs money or is irreversible.
• Actions that are dead until hydration. A button that navigates via a click handler does nothing until the JavaScript loads, which on a heavy page is a real window with a real conversion cost.
• Responsive overlap that makes a control unclickable while every desktop capture looks perfect.
• Empty states and long-content states. Real data breaks layouts that lorem ipsum never touches.
• The second-attempt path. Retry, resend, back button. A frontend that matches routes off the path during render with no history listener means the browser back button does nothing — which on a signup funnel is a conversion bug, not a nit.

The mitigation is not more unit tests. It is a short, fixed manual pass on the paths that carry money or identity, run before each release, by a person, with real credentials.

Bonus: Risk-Weighted Test Strategy — What to Test First

Coverage percentage is a bad target because it treats every line as equally worth protecting. It is not. A useful strategy ranks by consequence, and the ranking is short.

Tier one — test exhaustively, including the unhappy paths. Anything that moves money, changes who someone is, or deletes something. Payment intent creation, subscription state changes, authentication and authorization, account deletion, permission checks. These get integration tests, not unit tests, because the bugs live between the pieces.

Tier two — test the contract. Anything another system depends on: API responses, webhook payloads, database migration shapes, exported types. The test here is that the shape does not change without someone noticing.

Tier three — test the logic that is genuinely tricky. Date arithmetic, pricing rules, state machines, anything with a table of cases in the requirements. Table-driven tests, one case per row, and the table lives next to the requirement.

Tier four — everything else. A smoke test that it renders is frequently enough. Chasing coverage here is where teams spend most of their testing budget for almost none of the return.

Two rules that keep this honest:

• A bug that reaches production earns a test, always. Not to prevent that exact bug — it is already fixed — but because a bug that escaped is evidence about where your suite is blind, and the tier ranking should be updated, not just the file.
• If a tier-one path has no test, it outranks every other testing task on the board. Including the refactor someone wants to do first.

Bonus: Mutation Testing, and How It Lied

Mutation testing — deliberately breaking your code to check whether a test notices — is the best available answer to "is this coverage real." It has its own failure mode, and we hit it.

Testing a webhook retry drain, a mutation removed a null-exclusion guard from a database query. The mutant was reported as killed, so the guard read as genuinely covered.

It was not. The delivered-record fixture had no payload, so removing the guard caused the delivery function to throw, and the surrounding error-isolation try/catch swallowed the exception. The mutant died from an unrelated error, in the wrong place, for the wrong reason. The guard itself was never actually exercised.

Two lessons:

• A killed mutant tells you a test failed, not why. Where isolation try/catch blocks exist, a mutant can die from the wrong error and still count.
• Fixtures need to be realistic in the fields the code under test reads. A fixture missing a field the happy path requires converts every mutation in that area into an exception test.

Bonus: Fixtures, Test Data, and the Realism Floor

Most tests that pass while the code is broken pass because the fixture was too clean.

• Fixtures must be realistic in every field the code under test reads. The mutation-testing failure above happened because a delivered-record fixture had no payload, so removing a guard threw an unrelated error that a try/catch swallowed. The fixture was not wrong in an obvious way; it was missing a field the code path required, which converted a logic test into an exception test.
• Build fixtures from a factory with sensible defaults, then override only what the test is about. A test that constructs an object with twelve fields is a test where the reader cannot tell which field matters.
• Include at least one deliberately hostile fixture per tier-one path. The 90-character name, the emoji, the null middle name, the address with a newline in it, the amount of exactly zero.
• Never share mutable fixtures between tests. A test that passes alone and fails in the suite, or vice versa, is almost always this, and the debugging cost is out of all proportion to the fix.
• Seed data for end-to-end runs is code, not a database dump. A dump drifts from the schema silently, and its provenance is unknowable within a month.

The realism floor is the point where the fixture is close enough to production that a defect would show. Below that floor, tests are decorative regardless of how many there are.

Bonus: The TDD Loop We Actually Run

Not the ceremony version. Four steps, and the discipline is in the second one.

1. Write the failing test first, against the requirement, not the implementation. This ordering is the entire defence against the Phase 1 failure — a test written after the code tends to describe the code.
2. Run it and watch it fail for the reason you expect. A test that passes before you write the code is testing nothing. A test that fails for a different reason than expected is testing something else. This step catches both, and it is the step people skip.
3. Write the smallest thing that makes it pass.
4. Refactor with the test green, then run the whole suite, not just the file.

Two amendments from practice:

• Integration over unit, where money or identity is involved. A pure function extracted purely for testability moves the bug into how it is called, and that call site now has no test.
• The interface is the test surface. If something is hard to test, that is information about the interface, not a reason to add mocks.

Bonus: Flaky Tests Are a Signal, Not Noise

A flaky test is usually treated as an annoyance to be retried away. It is more often a real race condition that your users will hit and you will not be able to reproduce.

• Retrying to green is how a real bug becomes permanent. Automatic retries in CI convert an intermittent product defect into an invisible one. If retries exist, they must be reported, not silent.
• Quarantine, do not delete. Move a flaky test to a separately reported job, so it is visible without blocking. A deleted flaky test is a lost bug report.
• The three usual causes, in order of frequency: shared state between tests, real time, and real asynchrony. Shared state is a fixture problem. Real time is a Date.now() problem and the fix is injecting a clock. Real asynchrony is the interesting one, because it usually means the code under test genuinely has a race.
• Flakiness that only appears in CI is an environment difference worth naming — parallelism, machine speed, timezone, locale. All four have produced real production bugs for us after being dismissed as CI noise.

Track flaky rate as a number. A suite where three percent of runs need a retry is a suite the team has already stopped believing, and everything else in this guide stops working once that happens.

Bonus: End-to-End Tests That Survive Contact With a Real App

End-to-end suites rot faster than any other kind, and the rot is almost always the same four causes.

• Selectors tied to structure. A test that finds a button by its position in a DOM tree breaks on every layout change and teaches the team that end-to-end tests are unreliable. Query by role and accessible name — which has the useful side effect of failing when the element is not reachable by a screen reader either.
• Sleeps instead of conditions. A fixed wait is a bet on machine speed. Every one of them is either flaky on a slow runner or wasting seconds on a fast one. Wait for the condition: the element, the network response, the URL.
• Tests that depend on each other's data. Each test creates what it needs and cleans up after itself, or the suite can only ever run in one order and only on a clean database.
• Testing everything through the interface. End-to-end is expensive per assertion, so it should cover the paths from tier one of the strategy above and nothing else. Business logic gets tested directly. A twenty-minute end-to-end suite asserting form validation is a suite that will be disabled within a quarter.

Two things worth adding once the basics hold:

Run against a real deployed preview, not a local server. Every lie in Phase 5 was a local-environment lie. A preview build is the artifact you are actually shipping.

Capture a trace on failure. A screenshot tells you the end state. A trace tells you the sequence, the network, and the console, which is the difference between reproducing a failure and guessing at it.

Bonus: The Release Pass — Fifteen Minutes, By a Human

This is the fixed manual pass referenced in Phase 7. It is deliberately short, because a long checklist is one nobody runs. Every step names the credential and the expected result.

1. Sign up as a brand-new user, on the real deployed URL, with an email you can actually read. Confirm the email arrives, and click the link from the email client rather than pasting it. Both of the auth failures in this guide would have been caught here.
2. Sign in again as an existing user. The returning path is a different code path and it is the one nobody tests.
3. Complete one purchase end to end with a real card, at the smallest available price. Then double-click the pay button on the next one, deliberately, and confirm exactly one charge appears.
4. Check the thing they bought actually arrived — the download, the access grant, the email. Fulfilment is a separate system from payment and it fails separately.
5. Load the two highest-traffic pages at mobile width and tap the primary call to action immediately, before the page has settled. Anything dead until hydration shows up here.
6. Press the browser back button at three points in the main funnel.
7. Look at one page with real long content — the longest title, the fullest table.
8. Confirm error monitoring received something during the pass. If nothing arrived, either the pass was too gentle or monitoring is not wired up, and both are worth knowing.

Write the result somewhere durable with the date and the build. Eight steps, one person, fifteen minutes, and it catches the entire class of defect that Phase 7 describes.

Bonus: The Testing Readiness Checklist

• CI output is grepped for "failed to run", not only for "failed"
• A minimum total test count is enforced, so a silently non-running suite fails the build
• Every reported pass count has been reproduced in a clean clone
• Every script in the package file is known, and the safety-critical ones are wired into CI
• A sample of tests has been read for what they actually assert, not counted
• No test asserts current behaviour purely because it is current behaviour
• Single-file runs use an explicit path flag, not a name pattern
• Worktree directory names cannot collide with test or feature names
• The pipeline is known to build the application, or documented as not
• No gate is permanently red
• Local verification kills by port and polls, never sleeps
• Mobile is verified with emulated device metrics, not a narrow window
• Layout is verified by DOM geometry, not by looking at a screenshot
• A wedged dev server is ruled out before debugging a suspected regression
• Payment paths are covered by session-creation tests that spend nothing
• Vendor-side defects are pinned as expected-to-fail, never skipped
• A fixed manual pass covers double-submit, pre-hydration clicks, empty states, and the back button
• Every manual QA step includes credentials, URL, and what correct looks like

Bonus: Tools and Resources Library

Runners and frameworks

• Vitest — the default for new work; fast, and the watch mode is good enough to leave running
• Jest — everywhere already; know that its path pattern matches the absolute path
• Playwright — end-to-end, device emulation, and the geometry assertions from Phase 5 in one tool
• Testing Library — queries by role and label, which pushes tests toward what a user can actually reach

Coverage and mutation

• Stryker — mutation testing for JavaScript and TypeScript, with the caveat from the bonus above
• c8 and istanbul — line coverage, useful mainly for spotting files that are not covered at all
• Codecov — trend over time is the useful signal; a single percentage is not

Verification and CI

• GitHub Actions — read the workflow file rather than trusting the badge
• Chrome DevTools Protocol — device metrics override, for real mobile measurement
• Percy or Chromatic — visual regression when the design system is stable enough for the diffs to mean something

Payments and integrations

• Stripe CLI — replay webhook events locally without a tunnel
• MSW — intercept at the network layer instead of mocking the client, so the test survives a client swap

Bonus: Recommended Reading

Start here

• Growing Object-Oriented Software, Guided by Tests — Freeman and Pryce. The source of the outside-in loop above
• Working Effectively with Legacy Code — Michael Feathers. Mostly about seams, which is mostly what testing is
• Unit Testing Principles, Practices, and Patterns — Vladimir Khorikov. The clearest treatment of what a good test actually is

Deeper

• Accelerate — Forsgren, Humble, Kim. The evidence that testing discipline and delivery speed are the same variable
• How to Measure Anything — Douglas Hubbard. Applies directly to the "is this coverage real" question
• The Goal — Goldratt. A novel about constraints, and the right lens for deciding what to test first

Worth following

• Kent C. Dodds — on testing behaviour instead of implementation
• Playwright's changelog — the tool moves fast and the new capabilities keep replacing custom harnesses
• Google Testing Blog — sparse, and worth reading back through

What's Next?

Guide 2 covers the design system, including the component rules that would have prevented the double-charge in Phase 7. Guide 3 covers production architecture. Guide 4 covers security and compliance, where several of the green-suite failures in Phase 1 turned into real vulnerabilities.

If you would rather have this installed than read about it, that is what we do.