Token

Posts that use this term

• Observability: what a production AI system actually logs

  Closing the AI Engineering series. Tracing prompts and tool calls, tracking spend and latency, and catching failures that don't throw an error.

• RAG done right: chunking and retrieval mechanics

  Naive RAG breaks in practice long before the model does. Chunking strategy and embedding choice decide most of it, and here's where top-k search quietly fails.

• Prompting as an engineering discipline

  A prompt that runs a thousand times a day needs to be treated like code, not a note you typed once. Few-shot, chain-of-thought, and structured output, with the trade-offs.

• Troubleshooting local LLMs (and how to keep up after this series)

  The full catalog of local-LLM failures: OOM, slow tok/s, garbage output, instruction drift, bad RAG hits, tool-call hallucination. Plus where to follow the field once you're on your own.

• Local RAG and embeddings

  Build a working local RAG pipeline in about 30 lines using nomic-embed-text, Chroma, and Llama 3.2. And why running it on your own machine beats the cloud for personal notes.

• Wiring a local LLM into the tools you already use

  How to point VS Code (Continue, Cline), web chat UIs (Open WebUI, LibreChat, Page Assist), and your own code at a local model using the OpenAI-compatible API. Swap cloud for local without rewriting anything.

• Your first local LLM, start to finish

  Install Ollama, pull Llama 3.2 3B, chat with it, hit its API, and fix the five things that break on a first install. You finish with a working local LLM.

• System requirements by OS for local LLMs

  What macOS, Linux, and Windows each need before you run a local LLM in 2026. Mac is the smoothest, Linux gives you the most knobs, and native Windows finally just works.

• Picking a local model by task

  The 2026 open leaders, sorted by what you actually want to do: coding, chat, the small-model crowd, structured output, vision, embeddings, and audio.

• Streaming, throughput, and the KV cache

  Why TTFT and tok/s are different numbers, why streaming feels faster than it is, and the KV cache that makes the 1000th token cost about the same as the first.

• Quantization, distillation, pruning: how a 140GB model fits on your laptop

  Three ways to shrink an LLM, and why one of them does almost all the work. What Q4_K_M actually means and what each shortcut costs you.

• The local-LLM vocabulary

  Parameters, B, dense vs MoE, base vs instruct, tokens, context windows, chat templates, GGUF, and quant suffixes. Read it once and any HuggingFace model card stops being scary.

• The pitch for local LLMs in 2026

  The case for running an LLM on the machine you already own. Privacy, no per-call cost, faster first token, no rate limits, and it works on a flight.

• What leaves your machine when you use AI

  What providers actually see, log, and keep when you call an LLM API in 2026. What "we don't train on your data" really means, how free and paid tiers differ, and when local is the only safe choice.

• LLM API bills, and why a token costs what it costs

  How input and output tokens get priced, why output runs 5-6x more, and how prompt caching cuts the input bill by 10x. Plus the hidden costs that ambush people.

• The runtimes: llama.cpp, Ollama, LM Studio

  llama.cpp is the engine. Ollama and LM Studio wrap it. What each one does, when to reach for which, and why the OpenAI-compatible APIs are mostly but not entirely interchangeable.

• Why Apple Silicon punches above its weight on local LLMs

  Unified memory lets the GPU see all of RAM. Here's why that beats a discrete-GPU PC past 32B parameters, what fits in 16/32/64/128/192GB, and where Apple Silicon still loses.

• What it takes to run a model on your own machine

  Why VRAM is the one number that decides whether a local LLM runs, what quantization really does to a model file, and the hardware ladder from an 8GB laptop to a 192GB workstation.

• The major LLMs in 2026

  A field guide to the closed frontier models and the open weights you can actually run. What the "B" numbers mean, and which size fits your machine.

• Where AI actually runs: cloud, local, edge

  When you use AI, a model file is sitting on a real machine. There are only three places it can be, and which one decides almost everything else.

• Prompt, RAG, fine-tune: three ways to shape a model

  Three levers for shaping what an LLM does: prompting (ask better), RAG (give it the right context), fine-tuning (change the weights). What each costs, what each fixes, and how to pick.

• RAG: giving a model memory it doesn't have

  RAG is the pattern of fetching relevant text from a search system and putting it in the LLM's context window before asking your question. Not magic, not fine-tuning, just better prompts.

• The context window, and why models hallucinate

  An LLM only sees a fixed-size slice of text at a time. When it doesn't know something, it predicts anyway. That's a hallucination, not a bug.

• From models to LLMs

  An LLM is one kind of ML model, trained on text, predicts the next token. That single trick at scale gets you ChatGPT, and also explains where it breaks.

• How a model learns: training and inference

  Training is the expensive one-time event where a model's numbers get tuned. Inference is the cheap repeated use afterwards. The gap in cost is enormous, and it shapes the whole industry.

• Install the Anthropic SDK

  Install the official Claude SDK for Python and Node, set your API key the safe way, and prove it works with a one-line call.