Your LLM Broke in Production and You Have No Idea Why
A customer opens a ticket. Your support bot told them something false, confidently, in a tone that sounded exactly like every correct answer it's ever given. You go to find out why. And you can't.
Not "it takes a while." You can't. The prompt template has been edited four times this month by three different people. The retrieval index rebuilt twice since the conversation happened. The model itself might be a different snapshot than the one running when the ticket was filed, because someone bumped a version pin in a dependency update that nobody flagged as model-relevant. You have logs. You have the final output. You do not have the chain of custody that produced it, and without that chain, "why did this happen" isn't a hard question — it's an unanswerable one.
This is the actual state of LLMOps at most companies running language models in production in 2026, and it's a strange place to have landed, because the software industry already solved this exact class of problem once.
MLOps Already Fixed This. LLMOps Threw It Away.
Data lineage tooling — the ability to trace a model's output back through the exact training data, feature version, and code path that produced it — became table stakes in MLOps roughly a decade ago. Tools like MLflow and Weights & Biases built entire product categories around answering "what changed, and when, and by whom" for traditional ML pipelines. It wasn't a nice-to-have. It was the thing that made production ML trustworthy enough to bet a business on, because the alternative — a model that occasionally does something wrong for reasons nobody can reconstruct — is not a system, it's a liability with a UI.
LLMOps arrived and, instead of inheriting that discipline, built something almost entirely oriented around the wrong stage of the pipeline. Walk through the current tooling landscape and you'll find beautifully designed prompt playgrounds, slick evaluation dashboards that score outputs against rubrics, and no shortage of vendors promising "observability" that turns out to mean a trace viewer showing you the request and the response. What's missing, consistently, across nearly every LLMOps stack I've looked at, is the boring middle: a durable, queryable record of exactly which prompt version, which retrieved context, which model checkpoint, and which system-prompt override combined to produce a specific output at a specific timestamp. Evaluation tells you a model is roughly 87% good on a benchmark. It tells you nothing about the one output that just cost you a customer.
Why This Gap Exists — And Why It's Not an Accident
The honest explanation isn't incompetence, it's incentive. Evaluation dashboards are demoable. You can screenshot a benchmark score improving quarter over quarter and put it in a board deck. Lineage tooling is invisible infrastructure — it does nothing for you until the one day something breaks, and then it's the only thing that matters. Every team under deadline pressure rationally underinvests in the thing that only pays off during an incident, and generative AI teams are under more deadline pressure than almost anyone else in the industry right now, shipping features against a hype cycle that doesn't wait for infrastructure maturity.
There's a second, more specific reason it's worse for LLMs than it was for traditional ML. A classic ML model's inputs are structured — a fixed feature vector, versioned in a table. An LLM's effective "input" is a live composite: the system prompt, the user message, retrieved documents from a vector index that gets re-embedded on its own schedule, and sometimes a chain of tool calls that each mutate context before the final generation happens. There is no single artifact to version. Lineage for an LLM output means capturing a snapshot of a moving system, not a row in a feature store, and most teams haven't built the infrastructure to do that because the traditional MLOps playbook didn't have to.
What Lineage Actually Requires for a RAG Pipeline
Concretely, tracing a single bad output back to its cause means being able to answer, after the fact, for that exact request: which version of the system prompt was active, which documents the retriever returned and in what order, what similarity scores those documents scored, which model checkpoint or API version served the generation, and whether any post-processing or guardrail step modified the output before it reached the user. Miss any one of those five and you're debugging with a blindfold on. Most teams I've seen have logs for maybe two of the five, usually the final prompt and the final output, which is roughly equivalent to debugging a car crash with only the dashcam footage of the moment of impact and none of the maintenance history.
The fix isn't exotic. It's the unglamorous work of treating every component of that composite input as a versioned artifact with a timestamp and a foreign key back to the request that used it — the same discipline data engineering applies to a batch pipeline, applied instead to a request that resolves in under a second. A few teams are doing this well; notably, it tends to be the teams that came from a strong MLOps background before LLMs existed, because they already had the muscle memory for "nothing ships without a lineage record," and they refused to drop it just because the artifact changed from a feature vector to a prompt template.
The Fix Nobody Wants to Build First
If you're setting up an LLM pipeline for production right now, the unsexy recommendation is to build the lineage layer before the evaluation dashboard, not after. It will not demo well. It will not show up in a launch announcement. But the evaluation dashboard tells you how your system performs on average, and averages are useless the moment an executive, a customer, or a regulator is asking about one specific bad output that already happened. When that moment comes — and for anyone running LLMs at real scale, it will — the only question that matters is whether you can reconstruct exactly what happened, or whether you're left explaining that the system is "generally pretty good" to someone who no longer cares about generally.
The industry keeps treating LLMOps as a new discipline that needs new tools invented from scratch. Mostly, it needs old discipline applied to a moving target. MLOps figured out that you can't trust what you can't trace. LLMOps is going to relearn that the hard way, one undiagnosable ticket at a time, until enough teams get burned that lineage stops being the thing everyone means to build eventually.
Cover photo by pipop kunachon via Pexels.