AI-Powered
Scaling.

Inside this guide you'll find the test that decides whether something should be a model call at all, the routing table we run in production, the fourteen days that cost us $498 and why every suspect was wrong, the crash loop that ran for two months, and the mechanical gate that catches an agent reporting success on work it never did.

Phase 1: Decide What the Model Is Actually For

The first and most expensive mistake in AI scaling is adding a model where deterministic code would have been better, faster, cheaper, and testable. Models are extraordinary at a narrow set of jobs and mediocre-to-dangerous at everything adjacent, and the difference is usually obvious once you name the job out loud.

The Three Jobs a Model Can Do

• Work that code genuinely cannot do. Reading unstructured text and returning structure. Judging tone. Summarising. Answering an open question against a document. This is the real category, and it is smaller than the enthusiasm suggests.
• Work code could do, but the rules are unknowable in advance. Classification where the categories keep shifting, extraction from formats you do not control. A model here is a hedge against a spec you cannot write, and it should be paired with a plan to replace it once the rules stabilise.
• A cosmetic layer over a deterministic engine. Narration, a subject line, a friendly summary of numbers a rules engine already decided. This is legitimate and it is where most of the cost savings live, because this job can run on the cheapest model available.

The Test Before You Add a Model

Four questions. If any answer is uncomfortable, the model is in the wrong place:

1. What happens if the output is wrong? If the answer is "money moves" or "a record is deleted," the model proposes and deterministic code decides. Always.
2. Is the output parsed downstream as structured data? Then constrain the shape at the API level rather than hoping, and validate before use. A model that must return one of five enum values cannot drift.
3. Could a lookup table do this? Frequently yes, and the table is instant, free, and testable.
4. What does this cost per call, and who can call it? Covered in Phase 3, and it is the question nobody asks until the bill arrives.

Phase 2: Routing — Tier by Job, Not by Prestige

Once you have more than one model call in a system, you have a routing decision, and defaulting every call to the largest model is the most expensive possible answer.

The Lanes

This is the actual routing table behind our own tooling, running through a proxy that classifies each request by intent and rewrites the backend before it is sent:

The Rule That Saves You

Any turn containing a tool call goes to the strongest model, before any other classification runs.

This single rule is most of the value. Smaller and non-native models flail at a tool-calling loop: they hallucinate tool names, emit malformed arguments, and lose the thread across a multi-step sequence. The failure is not a slightly worse answer, it is a session that spins and burns more tokens than the expensive model would have used. The classifier checks for the presence of tools first and skips everything else if they are there.

Two related traps from running this in production:

• Non-native lanes must strip extended-thinking blocks, or the provider rejects the request outright.
• Only the native model keeps prompt caching and extended thinking. Routing a long-context conversation to a cheaper backend can cost more, because you lose the cache and re-send the whole prefix at full price.

Case Study: Routing the CLI Off Metered Credits

A small configuration change with a large effect, and worth stating because it is invisible in any dashboard.

Our coding CLI sessions were billing metered API credits by default. Three lines of shell configuration — blanking the API key and repointing the base URL at the proxy — moved every CLI run onto the flat subscription instead. Same tool, same models for the lanes that need them, and the metered spend for that entire category dropped to zero.

The reason this matters beyond the money: when we later investigated a credit burn, every local tool was already provably not the cause, which eliminated the largest and noisiest category of suspect in one step. Routing is also an attribution tool.

Case Study: A Fleet That Costs Nothing to Run

Worth stating plainly because it is the strongest argument for routing that we have.

We run a set of always-on agents: outreach setters, engagement responders, ad-variant generators, content posters. Across 5,106 session files carrying 2,553 model-change events, every one of them was running a local open-weight model. One agent alone accounts for 13,171 session files.

The metered cost of that entire fleet is zero. Not reduced, zero. It is possible because the work those agents do is well-specified: read a state, apply rules, produce a short piece of prose, write a row. That is the third job from Phase 1 — a cosmetic layer over a deterministic engine — and it is the single largest category of agent work in most businesses.

The corollary is the useful part. When the credit investigation in Phase 3 began, this fleet was the loudest and most obvious suspect, and it was eliminated in one query because the routing was explicit and the session logs recorded which model actually served each turn. Log the model per call, or every cost investigation starts from zero.

Where Free and Local Models Are Actually Safe

We run several always-on agents whose model call is pure narration over a deterministic engine. Those run on free-tier or local inference and cost nothing.

The boundary is sharp, and worth writing into your own code review:

• Safe: the output is cosmetic or recoverable, there is deterministic logic underneath, and the caller already handles a throw. A narrator, a summary, a subject line.
• Not safe: the output is the product, moves money, or is parsed downstream as structured data. Free models rate-limit unpredictably and drift from format, and both failures are silent until someone reads the output.

Phase 3: The Cost Model Nobody Builds Until It Hurts

Case Study: $498 in Fourteen Days

We investigated roughly $498 of API credits burned over two weeks on our own infrastructure. The investigation is worth walking because every intuition was wrong.

Ruled out, all verified rather than assumed:

• The coding CLI. Already routed off metered credits, per Phase 2.
• The agent gateways. All of them. We scanned 5,106 session files carrying 2,553 model-change events and found zero calls to the metered provider — every agent session was running a local open-weight model. One agent alone had 13,171 session files.
• Two scheduled jobs. One fires every three hours and dies instantly on a missing module. It burns nothing and does nothing, which are two separate findings. The other's schedule was never actually loaded, so it had run 13 times and stopped weeks earlier.

Where it actually was: the production website. Seven public routes call a model, and one of them — the instant quote — has no authentication and no rate limit. A twelve-minute log tail caught roughly 26 requests to that route and 6 to the expensive contract route bunched into a single 75-second burst, on a page whose requests are button-driven. Organic use does not look like that.

The Four Levers

In the order they actually pay off:

1. Gate the endpoint. A model-backed route reachable by anyone is a denial-of-wallet vulnerability, and it will be found by automated scanning long before anyone with intent finds it. A session, a token, a challenge, or a hard provider-side budget cap.
2. Cut the output ceiling. Cost scales with output tokens far more than input. A 16,000-token ceiling on a route that produces a two-page document is a bet that nothing ever asks for the maximum.
3. Route down. Most calls do not need the largest model. Phase 2 is the mechanism.
4. Cache the prefix. A long system prompt or document sent on every call is the single biggest avoidable input cost, and prompt caching removes most of it. Structure prompts so the stable part comes first.

And the monitoring rule that would have caught all of this on day one rather than day fourteen: alert on spend rate, not on spend total. A daily total tells you after the fact. A rate alarm tells you during. Add per-key attribution while you are there, so the alarm names the culprit instead of the account.

Phase 4: Agents That Run Without You

An always-on agent is a piece of infrastructure, and it fails like infrastructure rather than like code. Nothing throws in your face. It just quietly stops being useful, or quietly becomes expensive, and both states can persist for months.

Case Study: The Crash Loop That Ran for Two Months

Four agent sub-gateways sat in a crash-restart loop for over two months. Each was configured to restart on exit, and each died immediately on a missing module, so the supervisor faithfully restarted them thousands of times a day.

The visible symptom was not the agents. It was the disk: each of their error logs had reached 582 MB, growing at roughly 800 bytes every three seconds. Nobody noticed until a build failed for lack of space.

Two failures compounding, and both are common:

• Restart-on-failure with no crash-loop backoff or alarm converts a hard failure into a silent one. The supervisor is doing exactly what it was told; the configuration is what is wrong.
• A scheduled job that fails identically every run is indistinguishable from one that succeeds, if nobody is checking the output rather than the exit.

In the same audit, a second job's schedule had never been loaded at all. It ran 13 times, stopped, and nothing anywhere noticed for weeks.

The Supervision Rules

• Every autonomous run writes a result somewhere a human will see. A row, a message, a file with a timestamp. "It ran" is not a result; "it processed 0 items for the 340th consecutive time" is.
• Crash loops need backoff and a ceiling. N failures in M minutes stops the service and alerts, rather than restarting forever.
• Rotate and cap logs before you need to. This costs one configuration line and prevents the failure above entirely.
• Every agent has a kill switch that does not require a deploy. A flag it checks at the start of each run.
• Every agent has an owner and a review date. Ours are reviewed when something breaks, which is how four of them ran broken for two months.
• Know your job's lifetime. A background task started inside an interactive session dies with that session. We launched a long install that way, saw the launch confirmation, and found later that the output file was completely empty and nothing had been installed. For work that must outlive the session, use a real scheduler.

Phase 5: "Completed" Is a Claim, Not a Fact

This is the phase that matters most, and it is the one nobody expects to need. The single most reliable behaviour of an AI agent is reporting that it finished.

Case Study: Four of Six Agents Signalled Done and Sent Nothing

On a six-lane codebase audit, each lane ran as a named background agent reporting to a lead. Four of the six signalled idle without ever sending their findings. The idle signal means "no more work queued," not "output delivered," and treating them as the same thing silently drops the agent's entire contribution.

A fifth lane produced a stranger failure. When re-prompted, it explained that it had been writing its report into its own reply body, which the lead cannot see. Its words: it was finished, it just never left the terminal. That is worse than silence, because the agent itself believed it had delivered.

A sixth quirk from the same run: one report arrived as three identical consecutive deliveries. A delivery glitch, not three reports, and re-synthesising it three times would have been the natural mistake.

Case Study: The Agent That Reported Success and Changed Nothing

A background agent was given one job: shrink a file below a size cap. It returned completed, with a confident result summary. The summary was actually a restatement of an unrelated task's output, and the file was byte-for-byte unchanged.

The same failure occurs without any agent involved. In an ordinary session, a final turn stated that a memory file had been updated with three specific findings. The transcript contains no write call that turn, and the file's modification time predated the session by five days. The primary model narrated an action it never took.

The Mechanical Gate

The pattern that makes fan-out safe: verify each worker's output with a check that is stricter than its own self-report, and that the worker did not write.

• Choose a checkable side effect before dispatching. A file that must exist, a byte count that must change, a row that must appear, a test that must pass.
• Run the check centrally, after all workers report, from the coordinator.
• Re-dispatch on failure rather than asking the worker whether it is sure.
• Never accept a summary as evidence for anything with a checkable side effect.
• Read the transcript when a result looks too fast. An agent that finishes a large task in six seconds usually did not do it, and its own account of why will be confidently wrong.

Phase 6: Fan-Out Without Collisions

Parallelism is where the throughput gain lives, and it is also where correctness quietly dies. The rules are few and they are absolute.

The Concurrency Rules

• One writer per working tree. Two agents editing the same checkout produce conflicts that present as logic bugs. Each writing agent gets an isolated tree and an explicit, non-overlapping list of files it owns.
• Read-only workers can run without limit. The constraint is on writes.
• Exactly one owner for shared manifests. Package files, lockfiles, and configuration are edited by one integrator. This single rule removes most parallel-agent merge pain.
• A child may drop capabilities but never add them. No worker gets more tools, more network, more spend, or more delegation depth than its parent.
• Check which repository your isolation actually targets. Our harness creates isolated trees under the current session's repository regardless of which repository the task names. Three agents dispatched to work on a sibling repository were silently mis-scoped; one spent 27 minutes producing genuinely good read-only research and could not commit a line of it.

Case Study: The Six-Lane Audit

The structure that has worked repeatedly for large read-only work, with each element there for a reason:

• Lanes with hard, non-overlapping scope. Frontend, backend, data, infrastructure, security, and trust. Each brief names the directories that lane owns and the ones it does not, with the sibling that owns them. Without that, three lanes audit authentication and nobody audits the build.
• A read-only guard as the first line of every brief. Do not edit, commit, or create files.
• An "already established, do not re-derive" block. Facts the coordinator has already confirmed, fed forward, so budget goes to new ground rather than re-proving old findings.
• Questions ranked by what the client is paying to learn, not a generic instruction to review the code.
• A human doing the things agents cannot. Driving the real application, reading a real inbox, clicking the real signup. The reading track cannot click a button; the human track cannot read 40,000 lines. Neither finds what the other finds.

Phase 7: Evaluation — Knowing It Got Better

Prompt changes feel like improvements. Without evaluation, that feeling is the only evidence you will ever have, and it is wrong roughly half the time.

The Golden Set

Twenty to fifty real inputs with known-good outputs, drawn from production rather than imagined. It does not need to be large; it needs to be real and fixed. Every prompt change, model change, and routing change runs against it before shipping.

Two rules that keep it honest:

• Every production failure joins the set. That is the entire mechanism by which the set becomes valuable, and it costs nothing.
• Never edit a case to make it pass. If a case is genuinely wrong, delete it and say so. Adjusting the expected output to match current behaviour is the same failure as writing a test against the implementation.

Using a Model as the Judge, and Its Three Failure Modes

An LLM judge is the only practical way to score open-ended output at volume. It has known biases and you should design around them rather than discover them:

• Position bias. Judges favour whichever candidate is shown first. Randomise the order, or run both orders and require agreement.
• Length bias. Judges reward longer answers. Constrain length in the rubric, or normalise for it.
• Self-preference. A judge favours output from its own model family. Use a different family for judging than for generating where you can.

And the structural rule: the judge scores against a rubric with named criteria, not a general impression. "Rate this 1 to 10" produces noise. "Does it cite a source, does it answer the question asked, does it avoid claims not in the document" produces a signal you can act on.

Phase 8: The Infrastructure Underneath

Model calls are slow, expensive, rate-limited, and occasionally fail. That combination makes them unlike almost any other call in your system, and the infrastructure has to reflect it.

• Anything user-visible and model-backed goes through a queue. A synchronous request holding a connection open for thirty seconds is an outage waiting for traffic.
• Every job is idempotent, keyed on the input. Retries are guaranteed, and a retry that duplicates work costs money twice and can double-send whatever the job produces.
• Timeouts everywhere, and shorter than you think. The default in most SDKs is far too long. A hung call that holds a worker is worse than a failed call that retries.
• Backpressure, not unbounded queueing. When the provider rate-limits you, shed or defer load rather than accumulating a backlog you will pay to process at 3am.
• Retry with jitter, and a ceiling. Synchronised retries after a provider blip produce a thundering herd against a service that is already struggling.
• Log the prompt, the model, the token counts, and the cost, per call. Without this, every question in Phase 3 is unanswerable. With it, they are a query.
• Version your prompts like code. A prompt is the highest-leverage, least-reviewed artifact in the system. It belongs in the repository, in review, and in the golden-set run.

Phase 9: Retrieval — When the Model Needs Your Data

Most useful AI features are not the model knowing things, they are the model reading your things. Retrieval is where that happens, and it fails in ways that look like model failures and are not.

The Failure That Looks Like a Hallucination

The most common retrieval bug in production: retrieval returns nothing useful, and the model answers anyway. It has no way to know its context was empty or irrelevant, so it produces a fluent, confident, entirely invented answer. The team concludes the model hallucinates and starts changing prompts, when the actual defect is two layers down in the search.

Three guards, in order of value:

• Check the retrieval before you call the model. If the best match scores below a floor, return "I don't have that" rather than calling the model at all. This one change removes most hallucination complaints.
• Require citations, and validate them. The answer must reference retrieved chunk ids, and code verifies those ids were actually in the context. A model that must cite cannot invent as freely.
• Log what was retrieved alongside every answer. When someone reports a bad answer, the first question is what the model was looking at, and without this it is unanswerable.

The Parts That Actually Determine Quality

• Chunking matters more than the embedding model. Split on document structure — headings, sections, list boundaries — not on a fixed character count that cuts sentences in half. Overlap adjacent chunks slightly. Carry the document title and section heading into the chunk text, because a chunk that reads "it must be signed by the Account Holder" is meaningless without knowing which document and which section it came from.
• Hybrid search beats pure vector search almost always. Vector search is weak at exact terms — product codes, error strings, names, version numbers — which are precisely the things people search for. Combine keyword and vector, then merge.
• Rerank the top results. Retrieve broadly, say fifty candidates, then rerank to the best five with a cross-encoder before they reach the model. This is usually the largest single quality jump available.
• Store metadata and filter on it first. Tenant, document type, date, permissions. Filtering before searching is faster, cheaper, and it is also how you avoid the worst possible retrieval bug, which is returning another customer's document.

Evaluating Retrieval Separately From Generation

They are two systems and they fail independently, so score them independently:

• Retrieval: for each golden-set question, is the correct chunk in the top five? That single number tells you whether generation is even being given a chance.
• Generation: given the correct chunks, is the answer right and grounded in them?

A team that only measures the end-to-end answer cannot tell a chunking problem from a prompting problem, and will spend weeks tuning the wrong one. We have watched exactly that happen, and the fix once measured was a chunking change that took an afternoon.

Start simple. A single Postgres table with a vector column handles far more than people expect, keeps your data in one place, and lets you filter on metadata with plain SQL. Reach for a dedicated vector database when you have measured a reason to.

Phase 10: Building a Custom Agent — The Anatomy

Everything above is about running agents well. This is about building one. We run a fleet of them, and every single one is the same six parts. Get all six right and the agent is boring, which is the goal.

The Six Parts

1. The soul. The instruction document that defines the agent: who it is, what it is allowed to do, what it must never do, where it escalates, and how it writes. Ours run to roughly six thousand characters and every one carries the same required sections — identity, scope, hard rules, escalation channel, and a tone section with an actual one-line sample of the voice. The sample matters more than the adjectives; "professional but warm" produces nothing, one real sentence produces a voice.

2. Tools, and only tools that exist. An agent should be given the smallest set of actions that lets it finish its job. Two rules from productising ours: connect only what is actually wired, and be honest about the rest. We support nine integrations; every other tool a role might want is declared explicitly as unsupported and shown as "coming soon" rather than offered and broken. Never sell a 404 connector.

The related defect is subtler and we shipped it twice: a soul that names a tool in its instructions which is neither connected nor declared unsupported. The agent then confidently plans work it cannot perform. Cross-check every tool named in the prose against the tools actually granted.

3. The trigger, chosen by what the agent must touch. Three mechanisms, three lifecycles, and picking wrong is a silent failure:

Most of our agents need local connectors and repos, so they run on a machine scheduler and the machine has to stay awake. That is a real constraint and it belongs in the design conversation, not discovered later.

4. State that outlives the process. Anything serverless will cold-start, so in-process memory is not state. Ours persist a JSON document to object storage or a database on every run, and read it back at the start of the next one. That single decision is what makes an agent restartable, debuggable, and idempotent. It is also where the range-query trap from Guide 3 lives: a due-work query on a nullable timestamp will happily match every already-completed record.

5. The gate. The most important design decision in the whole agent, and it is binary: does this thing act, or does it propose?

Our ad-generation agent runs daily, does the full pipeline, and then emails a preview and stops. Nothing spends money until a human clicks Approve. The approve link is signed, and a forged link is rejected. That gate is why the agent could be turned on months before anyone fully trusted it.

6. The surface. A health endpoint returning JSON is not a user interface. Every agent that a human is expected to supervise needs a page that lists recent runs newest-first, shows what was produced, and carries the approve and reject controls. We added one to the ad agent after being asked, reasonably, "is this a UI?"

Phase 11: Training an Agent — Which Knob Actually Moves Quality

"Training" is the word everyone reaches for, and fine-tuning is almost never the answer. There is a ladder, and you should exhaust it from the top.

The Ladder, Cheapest First

1. Give it better context. Most bad agent output is an agent that did not have the information. Retrieval, prior state, the actual record it is acting on. This fixes more than everything below it combined.
2. Give it better instructions. Specific, bounded, with the failure cases named. "Never send to a contact who replied in the last 30 days" beats "be respectful of the recipient."
3. Give it better tools. One action that does the right thing beats five actions it has to compose. If an agent keeps making the same three calls in the same order, that sequence should be one tool.
4. Give it real examples. Two or three examples of correct output, taken from production rather than invented, in the prompt. This is the highest-leverage change most teams never make.
5. Route the hard lane up. Per Phase 2. A bigger model for the ten percent of calls that need it, not for everything.
6. Fine-tune. Last. Justified when you need a consistent format or voice at high volume and have hundreds of real examples. It fixes style reliably and knowledge rarely, and it freezes you to a model version.

Writing the Soul Spec Once, Then Enforcing It

When we deepened 62 agent souls at once, the thing that made it work was writing a spec first and then verifying against it mechanically. Every file had to pass, independently of what any writer claimed:

• Valid JSON
• Soul body at or above a minimum length
• All required sections present
• Model id in an allowed set
• Identity fields byte-identical to the baseline
• Zero references to tools that are not actually granted

That last check is the one that catches the honesty bug from Phase 10, and it caught two more instances than manual review did.

Two Findings Worth Stealing

Self-report is scoped to what the reporter looked at, and is honest within that scope. Every writer truthfully reported its own files clean against a style rule. The package README, which no writer owned, violated the same rule throughout. Nobody lied; the scope of "all clear" was smaller than the scope of the claim. Run the gate over the whole artifact, not over the union of what each worker checked.

A threshold is not an approval gate. Reviewing those same agents, we found six that would send external messages with no human in the loop. One had a guard reading "never send to more than ten people at once" — which, read as code, explicitly licenses sending to nine unreviewed recipients every time. We rewrote all six so that anything externally visible and hard to reverse is drafted and held. There is no recipient count at which unreviewed sending becomes acceptable.

And a maintenance note that will apply to you within months: pinned model ids rot. Of those 62 agents, 54 were pinned to a model that had been superseded and 5 to another. Nothing broke, and the whole fleet was quietly a generation behind. Put the model id sweep on a calendar.

Phase 12: Productising Agents as Presets

The step from "we run agents" to "our customers run agents" is smaller than it looks and has three specific traps.

Presets, not blank canvases. Nobody configures an agent from an empty form. We ship 62 role presets, each carrying a production-quality soul, a model, and a pre-selected tool map. A customer picks a role and gets something that works, then edits. The generation itself was a fan-out: one agent per role, schema-validated output, deduped and emitted into a single source file.

Deterministic slugs. Every preset is addressable by a slug derived from its position by a shared helper, so the link generated on the marketing page and the lookup performed by the configuration page cannot drift. Two independent slug implementations is a bug you will find in production, not in review.

Honest capability. Covered in Phase 10 and repeated here because it is the one customers notice: a preset that lists a connector you have not built is a support ticket and a refund. Declare it unsupported.

Bonus: Where Not to Put a Model

The short list, each learned expensively by someone:

• Anywhere the output moves money without a deterministic check. The model proposes, code decides.
• As your only rate limiter, validator, or authorisation check. These must be deterministic and testable.
• In a loop with no ceiling. An agent that decides when it is done needs a hard cap on iterations, tokens, and wall-clock time.
• Where a lookup table would do. Cheaper, instant, free, and it never drifts.
• On a public route with no gate. See Phase 3, and the $498.
• Rendering unsanitised output into HTML, or interpolating unsanitised input into a prompt. Both are injection surfaces. User text reaching a model must be delimited and labelled as data; model output reaching a renderer or a privileged action must be checked first.

Bonus: The AI Scaling Readiness Checklist

• Every model call has a stated job from the three in Phase 1
• Anything that could be a lookup table is a lookup table
• Output shape is constrained at the API level and validated before use
• Routing tiers by job, and every tool-calling turn goes to the strongest model
• Cheap and local models are used only where output is cosmetic or recoverable
• Every public model-backed route has a gate and a per-call cost recorded
• No rate limiter depends on in-process memory on a serverless runtime
• Output ceilings are set to what the feature actually needs
• Prompt prefixes are stable and cached
• Spend is alerted on by rate and attributed per key
• Every autonomous run writes a visible result, not just an exit code
• Crash loops have backoff, a ceiling, and an alarm; logs are rotated and capped
• Every agent has a kill switch, an owner, and a review date
• Long-running work uses a scheduler that outlives the session
• Every agent report is verified against a checkable side effect the agent did not write
• One writer per working tree, one owner for shared manifests
• A golden set exists, every production failure joins it, and no case is edited to pass
• Judges are scored against named criteria, with position and length bias controlled
• Retrieval is checked against a score floor before the model is called at all
• Answers cite retrieved chunk ids, and code validates those ids were in context
• Permission filtering happens at the query, never as a prompt instruction
• Retrieval and generation are scored separately
• Model calls are queued, idempotent, timed out, and retried with jitter and a ceiling
• Prompts are versioned in the repository and reviewed like code
• Every agent's soul names its scope, hard rules, escalation channel, and a real tone sample
• Which config layer wins is written in the first paragraph of the runbook
• Every tool named in an agent's instructions is actually granted, or declared unsupported
• The trigger mechanism matches what the agent needs to reach
• State persists outside the process and survives a cold start
• The act-or-propose gate is a deliberate decision, and approval links are signed
• A supervision surface exists that is not a health endpoint
• Rungs 1 to 4 of the training ladder are exhausted before anyone says "fine-tune"
• The quality gate runs over the whole artifact, not the union of what each worker checked
• No approval gate is expressed as a recipient-count threshold
• Pinned model ids are swept on a schedule
• The degraded path is decided deliberately and tested

Bonus: Tools and Resources Library

Routing and gateways

• LiteLLM — the proxy behind the routing table in Phase 2; one API surface, per-request backend rewriting
• OpenRouter — many providers behind one key, including free tiers for cosmetic calls
• Ollama — local inference, and the reason our agent fleet costs nothing to run

Agents and orchestration

• Claude Code and the Claude Agent SDK — what we build our own agents on
• git worktree — the mechanism behind the one-writer-per-tree rule
• Temporal or Inngest — durable execution when a workflow must survive a crash mid-run

Evaluation

• Promptfoo — golden-set runs and judge scoring, in CI
• Braintrust and LangSmith — evaluation and tracing with a UI, when a team needs to look at the same data
• Ragas — retrieval-specific metrics, if you are doing retrieval

Observability and cost

• Helicone or Langfuse — per-call cost, latency, and token attribution, which is what makes Phase 3 answerable
• Your provider's own usage console, grouped by key — the reason to issue one key per surface

Infrastructure

• Upstash — queues and shared-state rate limiting that survive cold starts
• pgvector — retrieval without adding a second database
• Zod or equivalent — the validation layer between model output and everything downstream

Bonus: Recommended Reading

Start here

• Anthropic's "Building Effective Agents" — the clearest published statement of when an agent beats a workflow, and when it does not
• Simon Willison's blog — the most reliable running record of what actually works, including prompt injection
• Chip Huyen, "AI Engineering" — the closest thing to a textbook for the discipline this guide describes

Deeper

• Designing Data-Intensive Applications — Kleppmann. Phase 8 is mostly this book applied to a slow, expensive, flaky dependency
• Release It! — Nygard. Timeouts, bulkheads, and circuit breakers, all of which apply directly
• Eugene Yan's writing on evaluation — the practical treatment of golden sets and LLM judges

Worth following

• Provider changelogs and model cards — pricing, context, and capabilities change monthly and your routing table should change with them
• OWASP LLM Top 10 — the injection surface, catalogued
• Latent Space — the best signal-to-noise on what teams are actually shipping

What's Next?

This is the last guide in the playbook. Guide 1 covers product strategy and discovery, Guide 2 the design system, Guide 3 production architecture including repository structure and app-store releases, Guide 4 security and compliance, Guide 5 testing, and Guide 6 go-to-market.

If you would rather have this installed than read about it, that is what we do.