The Complete Digital Product Playbook
Full-Stack
Engineering.
Production-grade architecture from day one.
Production-grade architecture from day one.
Inside this guide you'll find the reversibility test we apply before choosing anything, four real production bugs that threw no exception, the two-command diagnosis for a credential your own API rejects, the bundle-cost measurement that saved a doomed refactor, and the shipping checks that catch what CI structurally cannot.
Most engineering decisions are cheap. You picked the wrong date library, you swap it in an afternoon. A handful of decisions are not cheap, and the entire discipline of architecture is knowing which handful.
The test we apply: if this turns out to be wrong in nine months, what does the fix touch? If the answer is one module, decide fast and move on. If the answer is every row in the database, or every deployed URL, or every integration partner, slow down and write the reasoning somewhere durable.
In practice, four decisions account for nearly all the expensive reversals we have seen:
Everything else — framework, styling approach, ORM, hosting region, test runner — is closer to reversible than it feels in the moment.
ConvergeFlow ran on a hosted auth provider's shared development instance, on a .vercel.app domain, well past launch. The visible symptom was small: the Facebook login button did not work.
That symptom looked like a config fix. It was not one, and the chain is worth walking because it is the same chain every time:
Development instances use the auth provider's own shared OAuth application for every social provider. You cannot supply your own client id and secret on a development instance — only on a production one. A production instance requires a domain you control with real DNS records, and a *.vercel.app host cannot become one. So "fix the Facebook button" expands to: acquire the domain, put it on the host, create the production auth instance, wait up to 48 hours for domain verification, re-issue OAuth credentials for every provider (Google breaks too — it was also riding shared development credentials), re-create the webhook endpoint and capture its new signing secret, migrate every user id in the database, and redeploy.
The migration is the part that silently loses data. Development and production instances are separate user stores, so every user receives a brand-new id. In this database that id appeared in two distinct shapes:
users, onboarding, leadBatchHistory, and user_profiles. These need copy-to-new-id-then-delete-original.A migration that only rewrites the field looks like it worked. It leaves every id-keyed document orphaned, and the user logs into an empty account. Nine Clerk users made this survivable. Nine hundred would not have.
.where('nextAttemptAt', '<=', nowIso)also matches every delivered row, because their value is null.where('nextAttemptAt', '>', '').where('nextAttemptAt', '<=', nowIso)an explicit lower bound excludes nullApplication bugs throw. Data-layer bugs return the wrong rows and let everything downstream succeed. That asymmetry is why data-layer review deserves disproportionate attention: the failure mode is not a stack trace, it is a partner receiving three thousand duplicate webhooks.
On Portable Tenant's backend we built a retry drain for outbound partner webhooks — find events whose next attempt is due, re-deliver them. The obvious query:
.where('nextAttemptAt', '<=', nowIso)Delivered records have nextAttemptAt: null. Firestore orders values by type first: null, then booleans, then numbers, then timestamps, then strings. A null sorts below every ISO string, so <= is trivially true for it. That query matches every already-delivered event in the collection.
Shipped, it would have re-POSTed every historical webhook to the partner — a live integration, with a real partner on the other end, receiving a replay of everything we had ever sent them. No error, no exception, a perfectly successful drain.
The fix is a lower bound that excludes null:
.where('nextAttemptAt', '>', '').where('nextAttemptAt', '<=', nowIso)WHERE x <= now silently drops nulls you wanted. Both are the same class of bug: the developer reasons about the values they were thinking of, and the database reasons about all the values that exist. Before shipping any range query, ask what this field is for the rows you were not thinking about.There is a second lesson buried in that fix, about how the test lied. A mutation-testing run reported the null guard as covered. It was not: the delivered-record fixture had no payload, so removing the guard made the delivery function throw, and the surrounding error-isolation try/catch swallowed it. The guard-removal mutant "passed" for entirely the wrong reason. A test that passes because a different error was swallowed is worse than no test, because it consumes the attention that would have found the gap.
subscription_status: 'active' and looked healthy.Testing checkout on two apps built on a low-code platform, we found both could take zero payments. Neither failure was visible from the UI.
The second app's failure was a row-level access rule: a club could be read only if its id was in {{user.data.member_club_ids}}. Two compounding faults sat behind it.
On the app side, four call sites wrote the array with a flat payload — updateMe({ member_club_ids: [...] }). That method PUTs its argument straight onto the user record, so a flat payload lands as top-level fields while user.data stays empty. Every reader looked in user.data. The correct shape nests it: updateMe({ data: { ...me.data, member_club_ids } }). The code read from one place and wrote to another, with a comment claiming it existed specifically to make the access rule work.
On the platform side, and this is the part no app change could fix: with the array holding exactly the right id, on a brand-new account with a fresh session, fetching the club still returned 404 and listing returned 0. The rule simply did not resolve. Only the platform console could change it.
The tell that something was already wrong here is worth memorising. The layout component retried four times with backoff and then substituted a stub object — a placeholder name and a fabricated subscription_status: 'active'. A retry-with-backoff-then-stub around a data read is a strong signal that someone hit an access-control problem and papered over it. It also meant the app looked healthy while displaying a fabricated subscription status.
club_id, the club was created, because you cannot obtain an id for a row that was never made. So the 404 is permissions, not absence. This one inference turns a day of guessing into a five-minute query.Code that moves money has a property no other code has: its failure mode is a customer who has paid and received nothing, and the discovery channel is an angry email rather than an alert.
The first of the two zero-payment apps failed on a single unsupported parameter. The checkout session creation passed statement_descriptor at the top level. Stripe does not accept it there — it belongs inside payment_intent_data as statement_descriptor_suffix. The result was Received unknown parameter, HTTP 500, on every payment attempt, on the live domain.
The bug had been live long enough that "we haven't had many sales yet" had become the accepted explanation.
Two things made it findable in minutes rather than days:
Reproduce against the real host with one request. Not a local copy, not a staging clone — the live domain, calling the function the pay button calls, with a real session token. Assumptions about "it's probably only the test environment" are how a revenue outage survives a week.
Prove checkout without spending anything. Creating a checkout session charges nobody; a card is only taken when someone submits the hosted page. So a test can create the session, assert the URL and currency, even load the hosted page, and then abandon. That covers the entire part of the flow holding the money logic, at zero cost. When the pay button itself is gated behind app state you cannot easily reach — one of these apps required ten profile views first — call the underlying function directly rather than trying to satisfy the gate.
A webhook signing secret is bound to one endpoint URL. Copying one site's secret to another site produces the worst available failure: payments succeed in the payment provider, the new site's webhook fails signature verification, and money is taken while the thing the customer bought is never provisioned. Silently.
Create a per-site endpoint and use the secret from that response. It is one API call, and it is the difference between a broken integration and a quiet revenue leak.
Two related rules that have each cost us something:
sk_live_ key and no test key stored anywhere. Wiring it into a new funnel means a completed checkout charges a real card. That sentence gets said out loud before the work starts, every time.200 and a resolving URL do not prove which account you acted on. We once created a live payment link through an integration platform and reported it as being on our account because the link resolved and reported live mode. It was on the platform's own connected account — a different business entity entirely. Verify the account id in the response before creating anything real: a product, a price, a payment link. Live mode proves some live account; it never proves which.Application code has a compiler, a linter, and tests. Configuration has none of these, which is why the most confusing production incidents are almost always configuration.
A partner escalated: their API key was being rejected by our staging environment. The key was correct. It had been correct for nine days.
The cause was structural. The deploy pipeline writes the entire runtime environment from a single stored secret — one blob, written to a .env file at deploy time. Which means every environment variable introduced into the codebase after that secret was last edited is silently undefined in production. No error. No boot warning. Code doing const key = process.env.PARTNER_KEY ?? '' simply takes the empty branch, forever.
The diagnosis is two commands and needs no cloud access at all:
gh secret list --repo <owner/repo>
# -> ENV_STAGING updated 2026-06-29
git log --format="%h %ad %s" --date=short -S "PARTNER_KEY" origin/staging | tail -1
# -> 2026-08-07Secret older than the variable means the variable has never been deployed. Conclusive. That pair of commands settled a nine-day-old partner escalation in one call, after a cloud CLI path was blocked by an expired token.
Two follow-ons that generalise:
Check every sibling variable added in the same change. Here the partner key, a callback URL, and a callback secret all landed on the same day, so outbound webhooks were dead too and nobody had noticed.
Know where the values live, separately from where the code runs. This service runs on one cloud provider and its environment values live in a source-control secret on another. "Aren't we using GCP?" has the answer "GCP is where it runs, GitHub is where the values live" — and the way to be sure is to read the deploy workflow, not to reason from the hosting bill.
The design lesson that shipped as the actual fix is the most portable thing in this guide.
A missing credential on your side and an incorrect credential on the caller's side must not return the same response. They had both returned 401, which is why nine days went by: from outside, a correct key, a garbage key, and no key at all all produced an identical 401. That identical response is itself the diagnosis — if all three match, the server has nothing to compare against, and the problem is yours.
Configured-but-wrong returns 401. Not-configured returns 503, and logs an error naming the missing variable. The caller learns something true, and your own logs tell you the answer without an investigation.
The most expensive performance work is the work aimed at the wrong thing, and the wrong thing is usually the file that looks scariest.
A marketing site shipped roughly 1.7 MB of client JavaScript. The obvious culprit was a 2,249-line landing page component marked 'use client', with nine useState and nine useEffect hooks. The brief was an islands refactor: split it up, stop hydrating everything.
We mapped every hook, planned the split, and then measured before touching anything.
The page's own code was 96 KB — 88 KB of that the page chunk itself. The other 1,605 KB was shared framework loaded on every page, including pages with no interactivity at all. The proposed refactor would have saved around 4% of total JavaScript, in exchange for restructuring the single riskiest file in the repository.
The actual weight was two dependencies sitting in the root layout's critical path: an analytics library at roughly 304 KB across two chunks, and an animation library at 118 KB. Deferring those is a low-risk change to two import statements. The file-size scare had hidden it completely.
Separate delivery cost from execution cost first. They are different axes and people conflate them constantly:
curl -s "$SITE/<page>" -o /tmp/a.html -D /tmp/a.head \
-w "html %{size_download}B ttfb %{time_starttransfer}s\n"
grep -i "x-vercel-cache\|age:" /tmp/a.headA cache hit with sub-second time-to-first-byte means delivery is fine and the complaint is about JavaScript execution, not the network.
Then diff production chunk manifests between two pages. This needs no local build at all — take the page under investigation and any unrelated page on the same site, ideally one with little interactivity:
curl -s "$SITE/<target>" | grep -oE '/_next/static/chunks/[^"]+\.js' | sort -u > /tmp/t.chunks
curl -s "$SITE/<other>" | grep -oE '/_next/static/chunks/[^"]+\.js' | sort -u > /tmp/o.chunks
comm -12 /tmp/t.chunks /tmp/o.chunks # shared: the framework cost
comm -23 /tmp/t.chunks /tmp/o.chunks # unique: the page's own costSum the sizes of each set. The ratio between them is the entire decision. If the shared set dominates, no amount of refactoring the page will help.
There is a companion reliability bug worth naming here, because it hides inside every "make this faster" job. A <button onClick> that navigates is dead until React hydrates. On a heavy page that is a real window — often a second or more on mobile — during which the primary call to action does nothing when clicked. The fix is a real <Link href> anchor, which works before any JavaScript runs. Be aware that this change breaks tests asserting on window.location.href, which is the small price.
A merged pull request is not a deploy, and a deploy is not a working page. The steps between them are where a surprising share of production incidents live.
Read your pipeline definition before trusting it. On one of our client repositories, npm run ci expands to type-check, lint, and test with coverage. Nothing in it builds the application. A build break is therefore invisible to CI and surfaces only on the hosting provider, after merge.
The inverse trap exists too: a local build can fail while the deployed build passes, because they use different bundlers. On the same repository, a plain webpack build fails on a generated-type constraint in a file untouched for seven months, while the real deploy script uses a different bundler that does not enforce that constraint — and builds the same commit fine. Confirm against a preview deployment before reporting a build break as a blocker.
Two more from the same repository, both worth checking on yours:
A collection of things that reported success and had not happened:
0. In a script, that is indistinguishable from success. Confirm the aliasing line actually appears in the output before trusting the URL.curl the live URL and check the status code, the title, the byte size, and one string you know is new. Four assertions, five seconds, and they are the only thing that actually proves the deploy landed. A command's exit code proves the command ran.auto aliaspicked Xcode 16; Apple has required 26+ since 28 Apr 2026. Build green, upload green, submit queued, never appeared in TestFlight.fmt header has no #ifndef guard, so it discards any -D flagREQUIRED_AGREEMENTS_MISSING_OR_EXPIRED. An unsigned Paid Applications Agreement. Account Holder only, web console only, no API, no rebuild helps.Everything in Phase 6 assumed you control the deploy. On iOS and Android you do not. A store release is not a deploy, it is a submission, and the difference is the entire difficulty: your artifact is handed to a third party who can reject it silently, for reasons that have nothing to do with your code, on a clock you do not set.
Two rules before any of the mechanics:
The clearest illustration we have of how store releases actually fail. Build 45 was green. Getting a build into TestFlight took until build 49, and every failure was a different category.
Build 46 — the silent rejection. Apple has required Xcode 26 or newer for App Store submissions since 28 April 2026. Our build profile pinned no image, so the build service's auto alias picked whatever it defaults to, which was an Xcode 16 image. The build compiled. It uploaded. The auto-submit queued successfully. And it never appeared in TestFlight, with no CLI-visible error at any point — the CLI at that version has no command to list submissions at all, so a rejected submission is invisible from the terminal. The only signal was Apple's email.
Build 47 and 48 — the fix that could not work. Pinning an Xcode 26 image traded a silent rejection for a hard failure. Xcode 26 ships a compiler that enforces stricter C++20 consteval rules, which the fmt library vendored by React Native 0.79 violates. The build now died in about two minutes with call to consteval function is not a constant expression.
Two escalating attempts at a compiler define both failed identically: first scoped to the offending pods, then broadened to every pod target plus an explicit flag. Reading the actual vendored header explained why. fmt decides that macro itself in an #if/#elif chain with no #ifndef guard, so it unconditionally redefines it and silently discards anything passed on the command line. The approach was structurally impossible, not mis-scoped — and three guesses were spent before anyone opened the header.
Build 49 — the fix that worked. Patch the vendored header directly in a post-install hook, after the dependency manager has written it. One gotcha that costs a build: the header is installed read-only, so the hook must make it writable first or the build dies with a permissions error on the CI machine.
And then the submission 403'd anyway. Build 49 compiled, uploaded, and every submission failed at the store handoff with FORBIDDEN.REQUIRED_AGREEMENTS_MISSING_OR_EXPIRED. That is not a build defect. The app has in-app purchases, so it was the Paid Applications Agreement, which Apple re-issues periodically and which silently rejects every upload until re-signed. No rebuild, no manifest key, and no build setting affects it. Only the Account Holder can accept it, only in the web console, and there is no API.
The parts that are genuinely different, rather than the parts every tutorial covers:
| iOS App Store | Google Play | |
|---|---|---|
| Identity | Apple Developer Program, $99/yr, and the Account Holder is a distinct role from Admin | Play Console, $25 once |
| Signing | Certificates and provisioning profiles, or let the build service manage them | An upload key you must never lose; Play re-signs with its own app signing key |
| Machine gate | Minimum Xcode version, enforced at submission, changes without warning | Target API level, enforced yearly, with a hard cutoff date |
| Test track | TestFlight, internal is instant, external needs review | Internal, closed, open, production, each with its own promotion step |
| Review | Days, human, and rejections cite guideline numbers | Usually faster, more automated, with a staged rollout percentage |
| The trap | Legal agreements silently 403 uploads | Staged rollout means "released" can mean 5% of users |
auto alias.js beside their .ts sources, one drifted and missing a security fix.ts/.tsx, so a violation in a .js twin passes itnpm test never invokes.github/ in 1,208 commits, beside 140 migrationsgit log --format='%an' | sort | uniq -c | sort -rn gives you the last one, and it belongs in every takeover assessment.Repository layout looks like taste and behaves like architecture. A good structure makes the right thing easy to find and the wrong thing hard to write. A bad one produces the failures below, all of which we have found in real audits, and none of which are about code quality.
Boring on purpose. Every deviation should be a decision someone can defend:
repo/
src/ application code, organised by feature not by type
tests/ mirrors src/ exactly, so a missing test is visible
scripts/ anything a human or CI runs by name
config/ environment and tool configuration
docs/
adr/ one file per irreversible decision
runbooks/ what to do when it breaks
.github/ CI. Its absence is itself a finding
CONTEXT.md the domain glossary the whole team uses
README.md how to run it, in under five commandsFour rules that do most of the work:
src/ by feature, not by type. A components/, hooks/, utils/ split means every change touches four directories and no directory tells you what the product does. Feature folders make the deletion test physical: you can delete a feature by deleting a folder.tests/ mirrors src/ exactly. Not because the runner needs it, but because the shape of the gap becomes visible at a glance.These are all from real audits of handed-over codebases, and every one of them was invisible to the test suite:
docs/handover/ and todo/. The todo/ runbooks were stale, and they told you to point the production environment at the staging database URL while pairing it with live payment keys. The project references were inverted relative to the authoritative docs. A new engineer reads whichever they find first. Name the authoritative directory in the README and delete or archive the other one..js files next to their .ts and .tsx originals, harmless only because the bundler happened to resolve TypeScript first. One of them had drifted: it was missing a security option and a bug fix its twin had gained. A resolution-order change would silently reintroduce a closed incident..ts and .tsx. A violation hidden in one of those 44 .js twins would pass the guard built specifically to prove such violations were absent.npm test invoked some of them. Fifteen failures lived in four separately named root scripts nobody ran, including the safety-envelope guards. List every script, and make CI run the ones that matter or delete them..github/ in 1,208 commits, alongside 140 migrations and 29 serverless functions. Every safety gate in that codebase was a sentence in a markdown file that a human had to remember to run.git log --format='%an' | sort | uniq -c | sort -rn takes two seconds.CONTEXT.md — the domain glossary. Ten to thirty terms, each one sentence, defining what a "project", a "run", a "tenant", or an "application" is in this product. It is the cheapest possible fix for the most expensive recurring problem, which is two engineers using the same word for different things. It also gives coding agents the vocabulary to name things consistently, which matters more every month.
docs/adr/ — one short file per irreversible decision, in the Phase 1 sense. What we chose, what we rejected, and what would make us revisit. Two paragraphs. The value is not the archive, it is that writing it surfaces the assumption you had not noticed, and it stops the same decision being relitigated every six months by whoever is newest.
CONTEXT.md and the ADRs are the two files worth loading into every agent's context, because they carry the decisions the code cannot tell you. And name worktree directories so they cannot collide with a test or feature name, or your single-file test run will quietly execute the entire suite.Most refactors move code without reducing complexity. The ones worth doing make a module deeper: more behaviour absorbed behind a smaller interface.
The vocabulary, borrowed from Ousterhout and used deliberately because consistent language is most of the value:
The deletion test is the working tool. Imagine deleting the module and inlining it everywhere. If complexity vanishes, it was a pass-through and deleting it is the refactor. If complexity reappears in five call sites at once, it was earning its keep.
Two corollaries we use constantly:
Every Varritech build now involves multiple agents writing code in parallel, and that changes the operational rules more than it changes the code.
Guide 4 covers security and compliance — the threat model, secrets, fail-open failures, and the erasure audit that finds the tables everyone forgets. Guide 5 covers testing, and specifically the ways a green suite lies. Guide 2 covers the design system that sits on top of this architecture.
If you would rather have this installed than read about it, that is what we do.
Seven guides in the playbook.
Product strategy, design systems, production engineering, security, testing, go-to-market, and AI scaling.