Observability: what a production AI system actually logs
Closing the AI Engineering series. Tracing prompts and tool calls, tracking spend and latency, and catching failures that don't throw an error.
Blog
All Posts
Posts on AI engineering, LLM systems, and software development.
Evals: how you actually know it got better
Feeling better isn't the same as measurably better. Offline eval sets, LLM-as-judge, human eval, and regression testing for prompts and RAG pipelines.
Guardrails: schema validation, filtering, and trust
A model's output isn't trusted just because it came from your own system. JSON-mode reliability, schema validation, content filtering, and basic jailbreak resistance.
Multi-agent systems: orchestration and handoffs
Adding a second agent doesn't halve your problems, it multiplies them. Supervisor patterns, handoffs, and the real cost of splitting one agent into several.
Agent memory: scratchpad vs long-term store
An agent forgets everything the moment its context window ends, unless something is built to save it. Short-term scratchpad vs a real long-term store, and what's actually worth keeping.
The agent loop: plan, act, observe
Most agent failures look the same: stuck calling the same tool forever. The plan-act-observe loop, why it gets stuck, and the fixes that actually work.
Beyond vector search: hybrid, rerank, and graph RAG
When plain vector search hits its ceiling, three upgrades actually help: hybrid search, reranking, and graph RAG. When each one earns its added complexity.
RAG done right: chunking and retrieval mechanics
Naive RAG breaks in practice long before the model does. Chunking strategy and embedding choice decide most of it, and here's where top-k search quietly fails.
Prompting as an engineering discipline
A prompt that runs a thousand times a day needs to be treated like code, not a note you typed once. Few-shot, chain-of-thought, and structured output, with the trade-offs.
From running models to building systems
Running a model and building with one are different skills. This post opens the AI Engineering series and lays out why the next nine posts exist.
Local Llms#13
Troubleshooting local LLMs (and how to keep up after this series)
The full catalog of local-LLM failures: OOM, slow tok/s, garbage output, instruction drift, bad RAG hits, tool-call hallucination. Plus where to follow the field once you're on your own.
Local Llms#12
Fine-tuning a model locally
When fine-tuning is actually the right call (it usually isn't) and how to pull off a LoRA run on a 16GB Mac, with a worked Llama 3.2 3B example.