LLM agents in production: from prototype to reliable system

You can build an agent demo in an afternoon. Building an LLM agent in production that won’t wake you up at 3 a.m. takes weeks. The gap between the two isn’t the model — it barely matters which one you pick — it’s everything wrapped around the call: how you orchestrate the steps, which tools you let it invoke, what it remembers between turns, what happens when the provider hands you a 429, and how you actually know whether the answer was any good. This article is what I’ve learned building LLM systems that have to keep working when nobody is watching.

What changes when you move from demo to production

In a demo, you control the input. You pick the prompt that looks good, run it three times, and record the run that comes out perfect. In production the input comes from a real user — sometimes acting in bad faith — and the system has to respond a thousand times in a row without degrading. That’s where four problems the demo hid come to the surface:

• Non-determinism. The same prompt gives different answers. What works “almost always” fails 5% of the time, and that 5% is your support queue.
• Real cost and latency. Every token costs money and time. An agent that chains six calls for a trivial task is unviable at scale.
• Provider failures. Rate limits, timeouts, outages. If your product depends on a single API, its downtime is your downtime.
• Actions with side effects. In the demo the agent “suggests.” In production the agent acts: it writes to a database, sends an email, deploys. A mistake stops being ugly text and becomes an incident.

Designing for production is, in essence, designing for these four failure modes from day one.

Orchestration: less autonomy, more control

The fully autonomous agent fantasy — give it a goal and it figures everything out — sells beautifully and works terribly. The freer you leave the reasoning loop, the more it drifts, the more it costs, and the harder it is to debug why it did what it did.

In practice, reliable systems look more like a state machine with an LLM in the nodes than a free-roaming agent. You define the possible steps and transitions; the model decides which branch to take and fills in the blanks, but it doesn’t invent the flow. That buys you something pure autonomy never will: traceability. When something fails, you know which node and why.

A heuristic I apply: start with the most rigid graph that solves the problem, and only add freedom where the rigid version falls short. A classifier that routes to three fixed branches solves more real cases than people expect, at a fraction of the cost of an agent that “reasons” on every turn.

The AI terminal on this very portfolio follows that philosophy: it isn’t an agent that can do anything, it’s an assistant with a bounded context (my projects and my stack, via RAG) and a single, well-defined job. Less surface area, fewer ways to break.

Function-calling: the contract with the real world

Tools (function-calling) are what turn a chatbot into an agent: they give it the ability to read fresh data and execute actions. They’re also the main source of subtle bugs.

Three rules that save me trouble:

1. Strict, validated schemas. The model will hand you arguments that almost match the schema. Validate with Zod (or Pydantic on a Python backend) before executing anything, and feed the validation error back to the model so it can correct itself. Don’t trust that the JSON comes out well-formed.
2. Idempotency and per-tool permissions. Reading is cheap and reversible; writing, running commands, or touching the network is not. Split tools by risk level and route the dangerous ones through a confirmation step — at least until you have statistical confidence in the behavior.
3. Descriptions as interface. Each tool’s description is its documentation for the model. An ambiguous description makes the agent call the wrong function. Treat descriptions as part of the code, not as a comment.

The underlying principle is plain old software design: small, specific interfaces. A tool that does one thing and does it clearly gets invoked correctly; a “Swiss-army-knife” tool with ten optional parameters confuses the model exactly the way it would confuse a human.

Memory: the problem isn’t remembering, it’s forgetting the right things

“Just give the agent memory” sounds trivial and isn’t. An LLM’s context is finite and expensive; you can’t stuff the whole history into every call. Useful memory is selection, not accumulation.

I distinguish three layers:

• Working memory: what’s happening in this conversation. It goes into the prompt in full while it fits; when it grows, it gets summarized.
• Long-term memory: facts that must persist across sessions (preferences, prior decisions). It lives in a vector store and is retrieved by relevance.
• Domain knowledge (RAG): the data the agent operates over. This isn’t the agent’s “memory,” it’s the source of truth it consults.

The piece that pays off most is RAG done well. In RecruitSecure AI — semantic candidate search that runs 100% in the browser with Transformers.js, a clear case of on-device AI — the trick isn’t the embedding model, it’s two-phase search: cheap boolean filters first (location, years of experience), and only then embeddings over the resulting subset. Retrieving just enough, exactly when you need it, is what makes a complex query respond fast instead of computing vectors over the whole corpus.

The same pattern applies to an agent’s memory: every retrieved token that doesn’t contribute is diluted signal, more cost, and more latency. The right question isn’t “what can I remember?” but “what’s the minimum I need right now?”

Guardrails: assume the agent will get it wrong

An LLM sounds just as confident when it’s right as when it’s making things up. That’s why guardrails aren’t a compliance add-on, they’re part of the system’s logic. They run in two directions:

• Input: sanitize what comes in. Detect prompt-injection attempts, trim sizes, reject out-of-domain requests before spending a single token.
• Output: validate what goes out before it reaches the user or fires an action. Deterministic checks — not “let another LLM review it.”

The AI terminal on this portfolio uses exactly this: a deterministic output filter that blocks off-tone responses before they’re shown. I don’t ask the model to “behave”; I verify its output with code that always returns the same result. The general rule: critical verification must never depend on another non-deterministic call. If your safety check can hallucinate, it isn’t a safety check.

Cost and latency: the cheapest agent is the one you don’t call

Every LLM call has a cost in money and in milliseconds, and agents chain calls. The levers that work best:

• Route by difficulty. Not everything needs the big model. A small, cheap classifier decides which cases deserve the expensive model. Most don’t.
• Cache what’s cacheable. Answers to frequent questions, already-computed embeddings, results from deterministic tools. A well-placed LRU cache removes a huge percentage of calls.
• Stream. It doesn’t reduce cost, but perceived latency collapses when the user sees tokens appear instead of a spinner.
• Bound the context. Fewer input tokens means less money and less latency on every turn. Context engineering is also cost engineering.

Resilience: no single provider is your only option

If your agent depends on a single API, its rate limit is your ceiling and its outage is your outage. The fix is a provider chain with fallback: if the primary fails or throttles you, you drop to the next one without the user noticing.

This isn’t theory. In DevFlow AI, the developer tooling suite, the AI runs over a free, resilient chain — Gemini Flash → Groq → OpenRouter → Pollinations — so the service stays up even if one of the links goes down. This portfolio’s terminal uses the same pattern, with Groq as primary and OpenRouter plus Pollinations as the safety net. The cost of building it is low; the cost of not having it is the bill you pay the day your only provider has a bad day.

On top of that, add the basics of any distributed system: aggressive timeouts, retries with backoff, and circuit breakers. An agent that sits waiting 60 seconds on a dead API isn’t resilient, it’s a bottleneck with good PR.

Evaluation: if you don’t measure it, it isn’t in production, it’s in testing

This is the part almost everyone skips, and the one that separates a toy from a system. You can’t improve what you don’t measure, and “I tried it and it works” is not a metric.

The minimum viable setup:

• An evaluation set. Real cases with their expected output. Every prompt or model change gets run against that set before deploying. Without it, every tweak is a blind bet.
• Traces of every run. Which tools it called, with what arguments, how many tokens, how long it took. When something fails in production, the trace is the difference between fixing it in ten minutes or in two days.
• Business metrics, not just technical ones. Resolution rate, human handoff, cost per task. Nobody cares about the model’s perplexity; the percentage of tickets the agent closes on its own — that they care about.

An agent with no way to verify its own work isn’t autonomous: it’s a plausible-text generator you still have to review line by line.

The full journey, not the demo

Taking an LLM agent to production is systems engineering with an AI component, not AI work with some software around it. Traceable orchestration, tools under contract, selective memory, deterministic guardrails, cost under control, multi-provider resilience, and continuous evaluation. Each of those pieces is the difference between an agent that helps and one that gets in the way.

It’s exactly the kind of problem I enjoy: taking a prototype that dazzles in a demo and turning it into something that holds up under real traffic, real errors, and real users without breaking.

If you’re taking LLM agents to production — or you have a promising prototype stuck at “almost” — let’s talk. I’ll tell you, no strings attached, where the potholes are before you hit them.