LLMs in production: the boring parts nobody demos

Anyone can build an impressive LLM demo in an afternoon. The hard work — the part that decides whether the thing actually runs on your customer traffic at 3am on a Saturday — is everything that doesn't fit in the keynote.

The demo-to-prod gap

The gap between a working notebook and a working product is rarely the model. It's the surrounding system: how you measure quality, how you handle the long tail of inputs, how you control cost, and how you fail safely when the model is wrong — because it will be wrong.

Evals are the new tests

Unit tests check that a function returns 4 when you ask for 2 + 2. LLM evals check that a fundamentally probabilistic system returns acceptable answers across a representative slice of the real world. The discipline is the same. The instinct that you need them is what most teams skip.

• Start with 30 hand-graded examples. Not 3,000. Thirty.
• Score on the dimension that matters to the business, not the one that's easy to measure.
• Run evals on every prompt change. Fail loud when a regression appears.
• Treat the eval set as a living asset. Every production incident adds one row.

Observability and cost

You cannot operate what you cannot see. Every LLM call in production should log the prompt, the response, the latency, the token count, and the cost — with a request ID that ties it back to the user action that triggered it. Without this, debugging is a séance.

Graceful failure

Models hallucinate. APIs go down. Rate limits hit at the worst moment. Design for it: validate structured outputs against a schema, fall back to a cached answer or a smaller model when the primary fails, and give the user a clear path when the system can't help — not a confidently wrong answer.

The teams that ship LLM features that stick are not the ones with the cleverest prompts. They're the ones who treat the language model as one component in a system that has to keep working when the model doesn't.