Observability: what a production AI system actually logs

Nine posts ago I told you about the small tool I built to handle X replies. It's still running. The reason I trust it isn't that it's never failed, it has. The reason I trust it is that I always know when it did.

That's observability: instrumenting a running system so you can tell what it actually did and why, after the fact, instead of finding out from a confused user or a bad outcome days later. Every post before this one was about building a piece of the system. This one is about knowing whether the whole thing, running unattended, right now, is actually healthy.

Why this matters more here than in normal software

A typical bug throws an error, a stack trace, a failed request, something loud. A lot of AI system failures don't. The model answers fluently, the JSON parses fine, the request returns 200, and the answer is just wrong, or based on the wrong retrieved chunk, or a tool call that technically succeeded but did the wrong thing. Nothing here trips a normal error monitor. That's exactly why logging has to be more deliberate than "log the errors," because the failures that matter most often aren't errors at all.

Tracing: reconstructing what actually happened

Tracing means recording the full sequence a request took, the prompt sent, any retrieval performed and what it returned, every tool call and its result, the final output, all tied together under one request ID. Without this, debugging a bad output means guessing at what the system might have done. With it, you can pull up the exact trace and see precisely which step went wrong, the retrieval that pulled the wrong chunk, the tool call with the malformed argument, the prompt that didn't include something it needed to.

This is the single highest-leverage thing to add if a system currently has none of it. Everything else on this list is easier once you can actually see what happened.

Spend and latency: the two numbers that quietly explode

Token spend and latency both have a habit of drifting upward slowly enough that nobody notices until a bill or a complaint arrives. Track both per request, not just as a monthly total, so a single expensive or slow request pattern shows up immediately instead of getting averaged away into a number that still looks fine.

A sudden latency spike is usually a real, findable cause, a model provider having a bad day, a retrieval step that started scanning more data than it should, a loop that's making more tool calls than the old version did. None of that is visible without per-request numbers to look at.

Catching the failures that don't throw an error

The hardest category, and the reason observability matters specifically for AI systems, is the silent wrong answer. A few things that actually help here: log enough of the trace that a human can spot-check outputs after the fact and catch drift before a user complains about it. Track the eval scores from the previous post over time in production, not just at ship time, since the regression testing discipline that catches a bad prompt change before shipping should keep running after shipping too. And when a user does flag something as wrong, treat that report as a signal to go pull the full trace, not just log the complaint and move on, because that one flagged case usually reveals a whole class of similar failures hiding in the traces nobody looked at.

Closing the series

Ten posts, start to finish. Post 1 made the case that running a model and building a system with one are different skills. Post 2 covered prompting as something you version and test, not type once. Posts 3 and 4 covered RAG, from naive vector search to hybrid, rerank, and graph variants. Posts 5 and 6 covered the agent loop and what an agent actually remembers between runs. Post 7 covered the real cost of splitting one agent into several. Post 8 covered not trusting a model's output just because your own system produced it. Post 9 covered how you actually know a change helped, instead of just feeling like it did. And this one is about knowing whether the whole thing is healthy right now, not just whether it worked in testing.

That small X-reply tool from post 1 is still the same idea running through all ten posts: the model was never the hard part. Everything wrapped around it, deciding, retrieving, remembering, coordinating, validating, measuring, watching, that's the actual discipline. Deployment at real scale and preference-based fine-tuning (RLHF, DPO) are the natural next things to learn once this foundation is solid, and they're a different series.

For now: go look at what your own system actually logs. If the honest answer is "not much," that's the one thing on this list worth fixing first.