From running models to building systems

I used to spend way too much time on X. Scrolling, reading threads, thinking of a good reply, then closing the tab without ever posting it. That kept happening until one weekend I decided to fix it myself. So I built a small tool that does the parts I kept skipping: it checks X for me, decides if a thread is actually worth replying to, writes a draft based on what the thread says, and sends that draft to my phone.

Teaching the model to write a decent reply, that part was actually the easy piece. The hard part was everything wrapped around it: deciding what's worth replying to in the first place, pulling in the right context before drafting, not writing something embarrassing, and getting all of that to work correctly every single time. That's a different skill than anything the last three series covered, and it's what this one is about.

That gap is what this series closes.

This is post 1 of 10 in the AI Engineering series. AI Foundations covered what a model actually is. AI Running and Local LLMs covered where a model runs and how to get one working on your own machine. All of that is real and useful, and none of it is the same skill as building a system that has to work every time, not just the one time you tested it in a chat window.

Running a model vs. building with one

Running a model means: you send it text, it sends text back. That's the whole contract. The local LLM walkthrough covers this end to end. Pull a model, chat with it, done.

Building a system means the model is one component in something bigger, and every other component can fail in its own way. A retrieval step can pull the wrong document. An agent can call the wrong tool, or the right tool with the wrong arguments. Output that's supposed to be JSON can come back looking like JSON and not actually parse. None of that shows up when you're chatting with a model in a terminal. All of it shows up the first week something runs unattended.

This is the real difference between "I understand how LLMs work" and "I can ship something that uses one." The first is what the last three series taught. The second is its own discipline, with its own failure modes and its own hard-won opinions, and almost none of it is about the model itself.

What the next nine posts cover

In order:

• Prompting as something you engineer and version, not type once and hope
• RAG, properly, past the single vector-search lookup from local RAG and embeddings
• Where naive retrieval stops working, and what replaces it
• The agent loop, and why it gets stuck
• What an agent actually needs to remember, and for how long
• When one agent stops being enough, and what breaks when you add more
• Why you can't trust a model's output any more than a stranger's
• How you actually know a change made the system better, not just different
• What a system running in production has to log

There's no fine-tuning post in this list on purpose. Fine-tuning a model locally already covered LoRA and QLoRA properly, and this series sits one layer above the model, at the part that decides whether the model is actually useful to anyone.

The honest starting point

Most of what goes wrong in a system like this isn't the model being dumb. It's the scaffolding around the model being sloppy. A chunking strategy nobody tested. A tool call nobody validated. A change shipped because it felt better, with nobody checking if it measurably was. The model is usually fine. The system around it usually isn't.

That's what the next nine posts are actually about.